Interlaced-to-progressive video

ABSTRACT

Interlaced-to-progressive conversion with (1) 3-2 pulldown detection, (2) pre-filtering for field motion detection, (3) field motion detection with feedback, (4) field edge detection including direction angle detection by comparison of pixel differences with sets of more than pairs of pixels and a lateral edge detection probability by a lack of non-lateral angle detection, and (5) blending moving pixel interpolation and still pixel interpolation using field uniformity.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from the following provisional patentapplications all filed Oct. 17, 2005: 60/727,298; 60/727,304;60/727,362; 60/727,369; and 60/727,818.

BACKGROUND OF THE INVENTION

The present invention relates to digital video signal processing, andmore particularly to interlace-to-progressive conversion.

For moving picture systems, interlaced video format is widely used toreduce data rate. The interlaced video format is illustrated in FIG. 15a, where t is the time, x horizontal position, and y vertical position.A video signal consists of a sequence of two kinds of fields. One of thetwo fields contains only lines with even line numbers, i.e., y=even(“Top Field”), and the other field contains those with odd line numbers(“Bottom Field”). For example, NTSC interlaced television sets display60 fields per second.

FIG. 15 b shows a progressive video format, which recent high-quality TVsets use to display video programs. In the progressive format, a framewhich contains all of the line data (progressive frame) is displayed ata time. NTSC progressive TV sets display 60 progressive frames persecond. Most video programs are broadcast in interlaced format. Theninterlace-to-progressive (IP) conversion is required to display the TVprogram on the progressive TV, i.e., skipped lines in interlaced fieldsneed to be filled in by interpolation.

Interlace-to-progressive conversion appears in various systems asillustrated in FIGS. 16 a-16 c; and FIG. 14 a illustrates aninterlace-to-progressive converter. The systems may have digital oranalog video input or both, which may be encoded. The system may havedigital or analog video output or both. For the system of FIG. 16 a theanalog input is A/D (analog to digital) converted to get a digital inputsignal. The digital video input signal is stored in a memory. The storedvideo signal is sent to the IP converter block. An OSD (on-screendisplay) block may be placed before or after the IP converter, or theOSD block may be omitted. The input signal may also sent to the IPconverter directly. In the IP converter the input signal is digitallyprocessed by a digital circuit or a processor with program code. Thedigital video output of the IP converter is sent to the outside directlyor output as analog video out through a D/A (digital to analog)converter.

The system of FIG. 16 b has an encoded video stream (e.g., MPEG2, MPEG4,H.264, et cetera) input which is decoded by the decoder. The decodedvideo signal is stored in a memory. The stored video signal is sent tothe IP converter block. Again, an OSD block may be placed before orafter the IP converter, or the OSD block may be omitted. The decodedvideo data may also sent to the IP converter directly. In the IPconverter the input signal is digitally processed by a digital circuitor a processor with program code. The digital video output of the IPconverter is sent to the outside directly or output as analog video outthrough D/A converter.

The system of FIG. 16 c has input from any of encoded video, digital,video or analog video. The system has any of digital video, analogvideo, or encoded video output. The system has both decoder and encoder(e.g., decoder and encoder for MPEG2, MPEG4, H.264, et cetera). Theencoded video stream is decoded by the decoder. The decoded video signalis stored in a memory. As in the other systems, the analog input is A/Dconverted to get digital input signal. The digital input signal is alsostored in a memory. The encoder encodes the input video signal andoutputs encoded video stream. The stored video signal is sent to the IPconverter block. The decoded video data or digital video input signalmay also sent to the IP converter directly. An OSD block may be placedbefore or after the IP converter, or the OSD block may be omitted. Inthe IP converter the input signal is digitally processed by a digitalcircuit or a processor with program code. The digital video output ofthe IP converter is sent to the outside directly or output as analogvideo out through a D/A converter.

FIG. 14 a is a block diagram of a generic motion adaptive IP converter.The converter converts input interlaced video source to progressivevideo format that contains the original interlaced lines plusinterpolated lines. The frame buffer stores several interlaced fields.The motion detector detects moving objects in the input fieldspixel-by-pixel. The detector calculates the amount of motion(Motion_level) at every pixel location where the pixel data needs to beinterpolated. The more obvious the motion is, the higher Motion_levelbecomes. The still-pixel and moving-pixel generators interpolate pixelsby assuming the pixel being interpolated is a part of a still and movingobject, respectively. The selector/blender block selects or blends theoutputs of the still-pixel and moving-pixel generators usingMotion_level. When Motion_level is low, the output of the still-pixelgenerator is selected or the blending fraction of the output of thestill-pixel generator in the interpolated output data becomes high. Thestill-pixel generator is realized by the inter-field interpolator. Themoving-pixel interpolator consists of the intra-field interpolator andthe edge-direction detector. The edge-direction detector detects thedirection at the detection pixel in the pattern of an object in thefield and outputs the detected direction to the intra-fieldinterpolator. The intra-field interpolator calculates a pixel valueusing the detected direction, by interpolating pixels along in thedetected direction. (Without a direction, the interpolator could simplyinterpolate using the two closest pixels in the field.) The spatialresolution of the inter-field interpolator is higher than that of theintra-field interpolator. But when an object which includes the pixelbeing generated by interpolation is moving, i.e., Motion_level is high,the inter-field interpolator causes comb-like artifacts and the outputof the intra-field interpolator is selected or the blending fraction ofthe output of the intra-field interpolator is set as high.

FIG. 14 b shows a schematic diagram of an edge-direction detector. Theedge-direction detector consists of the “directional index generator”and the “direction determiner”. The directional index generatorgenerates index values for various edge angles. The direction determinerchooses the most probable direction using the output of the directionalindex generator. The directional index generator consists of many indexgenerators: one for the edge angle #0, one for the edge angle #1, and soforth.

FIG. 19 a shows an example flow of a traditional motion detection methodfor interlace-to-progressive up-conversion processing. First, letfield#1 be a target field to be up-converted to progressive format, anddefine two input pixel (luminance) data arrays in the two neighboropposite fields#0 and #2, in_y[0][x][y] and in_y[2][x][y] where x=0, . .. , IMAGE_SIZE_X−1; y=0, 2, 4, . . . , IMAGE_SIZE_Y−2 (top field) ory=1, 3, 5, . . . , IMAGE_SIZE_Y−1 (bottom field). To each pixel at (x,y)in the target field#1, two two-dimensional M×N pixel arrays,mot_y[0][i][i] and mot_y[2][i][j] (i=0, . . . , M−1; j=0, . . . , N−1),around the pixel are extracted from the two neighbor fields of thetarget field as follows:mot_(—) y[k][i][j]=in_(—) y[k][x+i−(M−1)/2][y+2j−(N−1)](k=0, 2; i=0, 1,. . . , M−1; j=0, 1, . . . , N−1)

Second, compute the M×N array of absolute values of differences,abs_diff[i][j], from the two extracted M×N pixel arrays, mot_y[0][i][j]and mot_y[2][i][j], as follows:abs_diff[i][j]=1|mot_(—) y[0][i][j]−mot_(—) y[2][i][j]|(i=0, 1, . . . ,M−1; j=0, 1, . . . , N−1)

Third, each resultant absolute difference value in M×N arrayabs_diff[i][j] is multiplied by a weight factor weight_factor[i][j] ateach pixel with the weight factor based on spatial distance from thetarget pixel:weight_diff[i][j]=abs_diff[i][j]×weight_factor[i][j](i=0, . . . , M−1,j=0, . . . , N−1)Note that typically the sum of the weight factors is taken to equal 1and thereby normalize for the size of the M×N array.

Finally, the elements of the weighted M×N array are added together and arepresentative value sum_diff_area[x][y] is output as a degree of motionof the target pixel for motion detection processing:

${{sum\_ diff}{{{\_ area}\lbrack x\rbrack}\lbrack y\rbrack}} = {\sum\limits_{i = 0}^{M}{\sum\limits_{j = 0}^{N}{{{weight\_ diff}\lbrack i\rbrack}\lbrack j\rbrack}}}$(x=0, . . . , IMAGE_SIZE_X−1, y=0,2,4, . . . , IMAGE_SIZE_Y−2) or (x=0,. . . , IMAGE_SIZE_(—) X−1, y=1,3,5, . . . , IMAGE_SIZE_(—) Y−1)

The value sum_diff_area[x][y] is used in a decision circuit forinterpolation processing. If the value sum_diff_area[x][y] is low, thestate of the target pixel at (x,y) is taken to be “still” and the pixely′[x][y] is produced using temporal interpolation of the two pixels atthe same position as the target pixel at (x,y) in field#0 and field#2.For example,

${{{y^{\prime}\lbrack 1\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} = \frac{{{{{in\_ y}\lbrack 0\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} + {{{{in\_ y}\lbrack 2\rbrack}\lbrack x\rbrack}\lbrack y\rbrack}}{2}$And if the value sum_diff_area[x][y] is high, the state of the targetpixel at (x,y) is estimated as “moving”, and the pixel is interpolatedspatially using only the neighbor pixels in the same field. For example,

${{{y^{\prime}\lbrack 1\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} = \frac{{{{{in\_ y}\lbrack 1\rbrack}\lbrack x\rbrack}\left\lbrack {y - 1} \right\rbrack} + {{{{in\_ y}\lbrack 1\rbrack}\lbrack x\rbrack}\left\lbrack {y + 1} \right\rbrack}}{2}$For pixel values in the range of 0 to 255, a typical decision would besum_diff_area[x][y]≦50 indicates the state of (x,y) is still andsum_diff_area[x][y]>50 indicates the state of (x,y) is moving.

This basic strategy for interpolation method decision leads to aprogressive format image prog_y[x][y] as follows:

${{{{prog\_ y}\lbrack 1\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} = \left\{ {{\begin{matrix}{{{{in\_ y}\lbrack 1\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} & \left( {{y = 0},2,4,\ldots\mspace{14mu},{{{IMAGE\_ SIZE}{\_ Y}} - 2}} \right) \\{{{y^{\prime}\lbrack 1\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} & \left( {{y = 1},3,5,\ldots\mspace{14mu},{{{IMAGE\_ SIZE}{\_ Y}} - 1}} \right)\end{matrix}\mspace{20mu}{or}{{{{prog\_ y}\lbrack 1\rbrack}\lbrack x\rbrack}\lbrack y\rbrack}} = \left\{ {\begin{matrix}{{{{in\_ y}\lbrack 1\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} & \left( {{y = 1},3,5,\ldots\mspace{14mu},{{{IMAGE\_ SIZE}{\_ Y}} - 1}} \right) \\{{{y^{\prime}\lbrack 1\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} & \left( {{y = 0},2,4,\ldots\mspace{14mu},{{{IMAGE\_ SIZE}{\_ Y}} - 2}} \right)\end{matrix}\left( {{x = 0},\ldots\mspace{14mu},{{{IMAGE\_ SIZE}{\_ X}} - 1}} \right)} \right.} \right.$This traditional motion detection scheme has the following weak points:

-   -   1) A simple expansion of this method for more accurate motion        detection consumes memory work space for field buffering, its        access bandwidth on the memory bus and computation load for        calculating the motion degree is proportional to the number of        fields to be stored.    -   2) It is difficult to detect especially fast motion and periodic        motion in interlaced image sequences.

FIG. 19 b shows a typical interlaced image sequence which causes motiondetection loss as described in 2). In this example sequence, arectangular box is moving at a constant speed from left side to rightside. Let the value of white pixels in all fields be 255, and that ofblack pixels be 0. And let the size of the extracted arrays be M=5, N=3.Under these conditions, the degree of motion of a pixel (x0,y0) at thecenter of target field#1 using above traditional detection method isconsidered. First, the two two-dimensional M×N pixel arraysmot_y[0][i][j] and mot_y[2][i][j] (i=0, . . . , 4; j=0, 1, 2) areextracted as follows:

${{{{mot\_ y}\lbrack 0\rbrack}\lbrack i\rbrack}\lbrack j\rbrack} = \left\{ {{\begin{matrix}255 & \left( {{i = 0},4} \right) \\0 & ({otherwise})\end{matrix}\left( {{j = 0},\ldots\mspace{14mu},2} \right){{{{mot\_ y}\lbrack 2\rbrack}\lbrack i\rbrack}\lbrack j\rbrack}} = \left\{ {\begin{matrix}255 & \left( {{i = 0},4} \right) \\0 & ({otherwise})\end{matrix}\left( {{j = 0},\ldots\mspace{14mu},2} \right)} \right.} \right.$Note that left side of the rectangular box in field#0 and right side ofthe rectangular box in field#2 overlap each other at this target pixel(x0,y0). Therefore, all absolute values of difference between two setsof the extracted M×N pixel array abs_diff[i][j], are zero:abs_diff[i][j]=|mot _(—) y[0][i][j]−mot_(—) y[2][i][j]|=0 (i=0, . . . ,4, j=0, . . . , 2)Hence, the weighted absolute difference value weight_diff[i][j] and thesum of difference value sum_diff_area[x0][y0] also result in all zeroes:

weight_diff[i][j] = abs_diff[i][j] × weight_factor[i][j] = 0(i = 0, …  , 4, j = 0, …  , 2)${{sum\_ diff}{{{\_ area}\left\lbrack {x\; 0} \right\rbrack}\left\lbrack {y\; 0} \right\rbrack}} = {{\sum\limits_{i = 0}^{4}{\sum\limits_{j = 0}^{2}{{{weight\_ diff}\lbrack i\rbrack}\lbrack j\rbrack}}} = 0}$As a result that the degree of motion at (x0,y0), sum_diff_area[x0][y0],is zero, and the target pixel is treated as “still” and interpolatedusing temporal interpolation as follows:

${{{y^{\prime}\lbrack 1\rbrack}\left\lbrack {x\; 0} \right\rbrack}\left\lbrack {y\; 0} \right\rbrack} = {\frac{{{{{in\_ y}\lbrack 0\rbrack}\left\lbrack {x\; 0} \right\rbrack}\left\lbrack {y\; 0} \right\rbrack} + {{{{in\_ y}\lbrack 2\rbrack}\left\lbrack {x\; 0} \right\rbrack}\left\lbrack {y\; 0} \right\rbrack}}{2} = {\frac{255 + 255}{2} = 255}}$Thus, in this example sequence, the interlace-to-progressive processingwith the traditional motion detection method causes noticeable noise atthe center of the up-converted progressive image.

Theater movies (film) provide 24 progressive format frames per second,and the 3-2 pulldown process illustrated in FIG. 18 a converts such asequence of progressive format frames into a sequence of interlacedformat fields with a field rate of 60 fields per second. Note that somefields are repeated, and that the overall process is periodic with aperiod of four progressive frames and ten fields (five top fieldsincluding one repetition and five bottom fields with one repetition).And precise detection of whether a received interlace format fieldsequence was originally generated as a 3-2 pulldown sequence can havegreat beneficial effect in the quality of several kinds of subsequentvideo processing of the field sequence, including IP conversion andvideo compression.

Note that 3-2 pulldown sequences have a repeated field every fivefields; therefore 3-2 pulldown sequences can be detected by detectingthis regular pattern (called a 3-2 pattern hereinafter). There are,however, several difficulties in accurate detection of the 3-2 patternin actual environments:

Difficulty #1: Some sequences may have small parts of image segmentswith different formats within them. A typical example is a 3-2 pulldownfilm material that is overlaid by a 60-interlace-fields-per-second telopfor subtitles. Thus the major portion of each of the images is 3-2pulldown, so very small field differences appear once every five fields.Handling them as a 3-2 pulldown sequence, however, will result inundesirable quality, especially at the part with the different format.

Difficulty #2: The 3-2 pattern becomes unclear when a 3-2 pulldownsequence is compressed by a lossy compression method such as MPEG fortransmission and/or storage. Transmission noise is also the source ofnoise if the transmission is performed in analog format.

Difficulty #3: The 3-2 pattern disappears in case of still sequences.

Difficulty #4: The 3-2 pattern becomes subtle if the area of movingobjects is relatively small compared to the whole image area, (e.g., ˜1%of an image is moving objects).

FIG. 18 b shows the block diagram of a typical 3-2 pulldown detector. Itcompares field differences directly to detect the timing when the(local) minimal difference arrives, then detects the 3-2 pattern byusing the result. This can detect well-ordered sequences, but it doesnot provide any solution to the difficulties listed above.

However, known methods for IP conversion still need improvement.

A general background discussion of interlace-to-progressive methodsappears in de Haan et al, Deinterlacing-An Overview, 86 Proceedings ofthe IEEE 1839 (Sep. 1998).

SUMMARY OF THE INVENTION

The present invention provides multiple IP methods such as: 3-2pulldown, pre-filtering for field motion detection, field motiondetection with feedback, field edge detection with differences of upperand lower multi-pixel patterns plus a lateral edge indicator using thenon-lateral edge detections, and blending moving pixel and still pixelinterpolations using field uniformity.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1 a-1 c illustrate a preferred embodiment 3-2 pulldown detectionmethod.

FIGS. 2 a-2 b interpret example data.

FIGS. 3 a-3 b illustrate a preferred embodiment interlace-to-progressiveconverter.

FIGS. 4 a-4 e show field notation.

FIGS. 5 a-5 d are an example.

FIGS. 6 a-6 b are a flow diagram and example of a preferred embodiment.

FIGS. 7 a-7 b show video coding functional blocks.

FIGS. 8 a-8 j illustrate a preferred embodiment edge detection.

FIGS. 9 a-9 c show field notation.

FIGS. 10 a-10 d show successive frames and fields of an example.

FIGS. 11 a-11 b illustrate the example with a preferred embodimentmethod applied.

FIGS. 12 a-12 d illustrate a preferred embodimentinterlace-to-progressive converter.

FIGS. 13 a-13 f are interlace-to-progressive conversion examples.

FIGS. 14 a-14 f show a generic interlace-to-progressive converter withedge detection and blending.

FIGS. 15 a-15 b illustrate interlace and progressive formats.

FIGS. 16 a-16 c show systems which use interlace-to-progressiveconversion.

FIGS. 17 a-17 b illustrate a processor and network communication.

FIGS. 18 a-18 b show 3-2 pulldown and a known detector.

FIGS. 19 a-19 b show prior art motion detection and an example.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

1. Overview

Preferred embodiment interlaced-to-progressive video processing methodsinclude 3-2 pulldown detection, pre-filtering prior to field motiondetection, field motion detection with feedback, field edge detectionfor moving pixel interpolation, and blending moving and still pixelinterpolations considering field uniformity.

Preferred embodiment systems such as digital TV, DVD players, DVDrecorders, cellphones, PDAs, notebook computers, etc., perform preferredembodiment methods with any of several types of hardware: digital signalprocessors (DSPs), general purpose programmable processors, applicationspecific circuits, or systems on a chip (SoC) such as combinations of aDSP and a RISC processor together with various specialized programmableaccelerators such as illustrated in the processor of FIG. 17 a. A storedprogram in an onboard or external (flash EEP) ROM or FRAM couldimplement the signal processing. Analog-to-digital converters anddigital-to-analog converters can provide coupling to the real world,modulators and demodulators (plus antennas for air interfaces) canprovide coupling for transmission waveforms, and packetizers can provideformats for transmission over networks such as the Internet asillustrated in FIG. 17 b.

2A. Detection of 3-2 Pulldown

FIG. 1 a illustrates a first preferred embodiment 3-2 pulldown detectionmethod consisting of two parts: a field difference detector and a 3-2pulldown detector; these two detectors are described in the following.

(a) Field Difference Detector

FIG. 1 b shows the field difference detector. The field differencedetector determines a representative value, which is used as a fielddifference by the following 3-2 pulldown detector, from two consecutivefields of the same parity (two successive top fields or two successivebottom fields). It is designed such that it can deal with Difficulties#1, #2 and #4.

The field difference detector operates as follows. Let I(t, x, y) be theluminance value at spatial coordinate (x, y) and time t. First performlow-pass filtering to cope with noisy input (Difficulty #2). Thisfiltering increases the accuracy of the detection. The first preferredembodiment uses a simple 2×2 averaging filter to get the filtered image,Î(t, x, y):Î(t, x, y)=Σ_((u,v)εBLPF) I(t,x+u,y+v)/|B _(LPF)|where B_(LPF)={(i,j)|0≦1i,j≦1} is the block of points for the averagingneighborhood, and |B_(LPF)| is the number of elements of B_(LPF);namely, 4.

For each pixel location, the neighborhood sum of absolute differences ofthe filtered images, d(t, x, y), is derived as follows withB_(diff)={(i,0)|−4≦i≦3} (this neighborhood is defined as a horizontalstrip to ease hardware implementation but other small neighborhoods alsowork):d(t,x,y)=Σ_((u,v)εBdiff) |Î(t,x+u,y+v)−Î(t−2, x+u,y+v)|The maximum value of d(t, x, y) over the image, denoted D(t), is used asthe representative field difference:D(t)=max_((x,y)) d(t,x,y)For example, if the luminance pixels in the fields at t and t−2 are allapproximately the same except in a small region (e.g., an 8×1 pixelstrip) where the difference averages 50, then D(t) will equal 8×50;whereas, the average difference over all pixels is close to 0. Thususing the maximum value of d(t, x, y) for the field difference insteadof the sum or average of absolute differences over all pixels of thefields helps for Difficulties #1 and #4.

Note that for an original sequence of progressive frames containingmotion, the fields of the 3-2 pulldown sequence as illustrated in FIG.18 a (numbered left-to-right starting with t=0 for the “1 top” field atthe left and continuing through t=11 for the “5 bot” field at theright), would have D(0), D(1), D(2), D(3) all large, D(4)≈0 (fields att=4 and t=2 are both “2 top”), D(5), D(6), D(7), D(8) all large, D(9)≈0(fields at t=9 and t=7 are both “4 bot”), D(10), D(11), . . . , alllarge; and so forth with periodicity of 5. In contrast, if the inputsequence of fields is not a 3-2 pulldown and there is motion in thescene, then all D(t) will be large. Similarly, if there is no motion inthe scene, then all D(t) will be small, for both 3-2 pulldown andnon-3-2 pulldown field sequences.

(b) 3-2 Pulldown Detector

FIG. 1 c shows the 3-2 pulldown detector which detects the 3-2 patternfrom the field difference values D(t) provided by the preceding fielddifference detector. It is designed to deal with Difficulties #2 and #3.The 3-2 pulldown detector outputs for field t, J(t) and φ(t), indicatewhether the field is part of a 3-2 pulldown sequence (if J(t)=1) and ifso its phase (φ(t)).

The values for thresholds and coefficients in the following example workfor 8-bit luminance input values; i.e., with I(t, x, y) in the range 0to 255.

First, for an input field difference, D(t), determine the maximum value,D_(max)(t), among the preceding field differences that must be small ifthe current input field is a repeated field:D _(max)(t)=max {D(t-i)|i=0, 5}Thus both the top field repeat and the bottom field repeat are needed.

Also determine the minimum value, D_(min)(t), among preceding fielddifferences that must be large if the current input field is a repeatedfield in a 3-2 pulldown sequence of images with motion (all fields looklike repeats if there is no motion):D _(min)(t)=min {D(t-i)|1≦i≦9, i≠5}

FIGS. 2 a-2 b illustrate typical field difference data and interpret thevariables D_(max)(t) and D_(min)(t) during operation of the 3-2 pulldowndetector. Indeed, when the current field is a repeated field, thenD_(max)(t) becomes very small (rectangular box near horizontal axiscontaining D(t) and D(t−5) in FIG. 2 a) while D_(min)(t) becomes large(rectangular box away from horizontal axis containing D(t−1), D(t−2),D(t−3), D(t−4), D(t−6), D(t−7), D(t−8), and D(t−9) in FIG. 2 a). The twoparticular D(t-j)s determining D_(max)(t) and D_(min)(t) are shown byarrows in FIG. 2 a. Conversely, when the current field is not a repeatedfield, then D_(max)(t) becomes large (D(t) and D(t−5) in two small boxesin FIG. 2 b) and D_(min)(t) becomes small (rectangular box extendingdown to the horizontal axis and containing D(t−1), . . . , D(t−9) inFIG. 2 b). Note that FIG. 2 b depicts the same data as in FIG. 2 a butone field earlier; and again the two particular D(t-j)s determiningD_(max)(t) and D_(min)(t) are shown by arrows in FIG. 2 b.

The preferred embodiment methods define the repeated-field detectionvariable, S(t), by the expression:S(t)=(α_(max)max{D _(max)(t), T _(min) }<D _(min)(t))^(D _(max)(t)<T_(max))where T_(max) is a threshold (=8×64), T_(min) is another threshold(=8×4), and α_(max) is a safety coefficient (=1.5). That is, S(t)=1 andindicates field t is a repeated field if both α_(max)max{D_(max)(t),T_(min)}<D_(min)(t) and D_(max)(t)<T_(max); otherwise S(t)=0.

T_(max) is used to avoid false detection as a repeated field for fieldsthat belong to completely non 3-2 pulldown sequences where all D(t) arelikely to be large. D_(max)(t) small requires both of the fielddifferences D(t) and D(t−5) be small.

Analogously, α_(max)=1.5 is used to avoid false detection as a 3-2pulldown repeated field for very noisy input (Difficulty #2). Noisyinput increases all of the D(t) and may make D_(min)(t) appear large asif field t were a repeated field in a 3-2 pulldown sequence; so requireD_(min)(t) to be relatively large compared to the noise-increasedD_(max)(t) by requiring α_(max)D_(max)(t)<D_(min)(t) for a repeatedfield detection. Indeed, when a 3-2 pattern is very noisy, the patternis likely not detected and α_(max) cannot help; whereas, if α_(max) issmall, a non 3-2 pattern might be detected as a 3-2 pattern becausethere is a possibility that a non 3-2 pattern has smaller D(t) values att=0, 5 by chance, so α_(max)=1.5 helps avoid this; and lastly, withα_(max) small a noisy 3-2 pattern might be detected with the wrongphase, so α_(max)=1.5 helps avoid this.

T_(min) helps α_(max) to work correctly in the case of very smallD_(max)(t) by also requiring α_(max)T_(min)<D_(min)(t) for a repeatedfield detection.

The 3-2 pulldown field phase, φ(t), which indicates a field's offsetfrom the presumed repeated fields, can simply count the number of fieldsafter the latest detection of a repeated field:φ(t)=0 if S(t)=1 or φ(t−1)=4=φ(t−1)+1 otherwiseThe φ(t−1)=4 alternative maintains the proper phase even if S(t)=0.

If a repeated field is not detected (i.e., S(t)=0) where it should be(i.e., φ(t) =0), the sequence can no longer be considered as a 3-2pulldown sequence. Therefore, an anti-3-2 pulldown signal, R(t), isdefined by the following expression:R(t)=(S(t)=0)^(φ(t)=0)^(D(t)>T _(still))where T_(still) is a threshold (=8×24). That is R(t)=1 when S(t)=0(indicates is not a repeat) and φ(t)=0 (should be a repeat) togetherwith D(t)>T_(still) (there is motion in the scene); otherwise R(t)=0.And R(t)=1 indicates that the sequence is not a 3-2 pulldown sequence.

The threshold T_(still)deters an R(t)=1 decision that the sequence isnot a 3-2 pulldown sequence when S(t)=0 is not accurate. Indeed, if t isa repeated field, then D(t)<T_(still) which prevents R(t)=1 even ifD_(min)(t) is small and makes S(t) =0; for example, a still scene in a3-2 pulldown sequence.

The final 3-2 pulldown detection result, J(t), is calculated as follows:

J(t) = 0 when R(t) = 1   = 1 when S(t) = 1   = J(t − 1) otherwiseThe current field is declared a part of a 3-2 pulldown sequence whenJ(t)=1, and vice versa. And the phase of the current field is given byφ(t).2B. Examples of 3-2 Pulldown

Consider some simple examples to illustrate the operation of thepreferred embodiment methods.

(1) First example is a video sequence with high contrast motion (i.e., a(small) bright object moving with a dark background) and minimal noise.Consider a 3-2 pulldown sequence as in FIG. 18 a withD(0)≈D(1)≈D(2)≈D(3)≈8×150, D(4) ≈0 (repeated top field),D(5)≈D(6)≈D(7)≈D(8)≈8×150, D(9)≈0 (repeated bottom field), D(10)≈D(11)≈.. . 8×150. (For a non-3-2 pulldown sequence, D(4) ≈D(9)≈8×150.)

First, for t=9, D_(max)(9)≈0 and D_(min)(9)≈8×150. SoD_(max)(9)<T_(max)(=8×64); and α_(max) max{D_(max)(9),T_(min)}≈1.5×max{0, 8×4}=8×6<D_(min)(9). Thus S(9)=1, and this impliesφ(9)=0, R(9)=0, and J(9)=1. Thus field 9 is declared part of a 3-2pulldown sequence and its phase φ(9)=0 indicates that it is a repeatedfield.

Next, for t=10, 11, D_(max)(t)≈8×150 and D_(min)(t)≈0. SoD_(max)(t)>T_(max) (=8×64); and α_(max) max{D_(max)(t),T_(min)}≈1.5×max{8×150, 8×4}=8×225>D_(min)(t). Hence, S(10)=S(11)=. ..=0. First, S(10)=0 implies φ(10)=1 and so R(10)=0; hence, J(10)=J(9)(=1). That is, field 10 is declared part of a 3-2 pulldown sequence andhas phase φ(10)=1. Similarly, S(11)=0 implies φ(11)=2 and thus R(11)=0,and J(11)=J(10) (=1). Again, field 11 is declared part of a 3-2 pulldownsequence and has phase φ(11)=2. Likewise, S(12)=S(13)=0 imply φ(12)=3,φ(13)=4, and R(12)=R(13)=0; hence, J(13)=J(12)=J(11)=1.

The field differences are approximately periodic with period 5, so thefield t=14 computations essentially repeat those for t=9 field, those oft=15 essentially repeat those for t=10, and so forth.

(2) Second example is a video sequence of either a still scene (i.e., nomoving objects) plus large noise or a moving object with low contrast(i.e., object and background of approximately the same luminance).Indeed, take D(0)=8×10, D(1)=8×5, D(2)=8×15, D(3)=8×20, D(4)=8×3,D(5)=8×10, D(6)=8×25, D(7)=8×15, D(8)=8×10, D(9)=8×3, D(10)=8×8,D(11)=8×15, . . . . Note that this could be a noisy 3-2 pulldownsequence with repeated fields 4 and 9.

Thus first consider t=9, D_(max)(9)=8×3 and D_(min)(9)=8×5. SoD_(max)(9)<T_(max)(=8×64); and α_(max) max{D_(max)(9),T_(min)}1.5×max{8×3, 8×4}=8×6>D_(min)(9). So D_(min)(9) is not enoughlarger than D_(max)(9) and T_(min) to indicate a repeated field; hence,S(9)=0. But R(9)=0 because D(9)<T_(still) (=8×24), and no change is madein the output: J(9)=J(8). (However, a change to D(1)=8×7, implies S(9)=1and this example would be like the first example.)

Similarly, for t=10, 11, . . . , D_(max)(t)=8×10, 8×25, . . . andD_(min)(t)≈8×3, 8×3, . . . . So D_(max)(t)<T_(max)(=8×64), and α_(max),max{D_(max)(t), T_(min),}=1.5×max{8×10+, 8×4}=8×15>D_(min)(t), and againS(10)=S(11)=0. Also, again D(10)<T_(still), D(11)<T_(still), . . . , soR(10)=R(11)=0 and no change in output: J(11)=J(10)=J(9) with the phaseincrementing and rolling over at 4. This persists until the occurrenceof a field with S(t)=1 to switch from J(t)=0 to J(t)=1 or with R(t)=1 toswitch from J(t)=1 to J(t)=0. So the declaration whether the inputfields are from a 3-2 pulldown sequence does not change with this kindof input which means for still scenes or hard-to-detect motion no newdecision is made.

2C. Alternatives for 3-2 Pulldown

The 3-2 pulldown determination could use lowpass filtering prior tofield differencing and maximum neightbor differences over a field.

For example, the lowpass filtering could use filters other than the 2×2averaging, such as a 3×2 averaging or a 5×3 with weights

$\begin{bmatrix}1 & 2 & 2 & 2 & 1 \\2 & 4 & 4 & 4 & 2 \\1 & 2 & 2 & 2 & 1\end{bmatrix}/32.$

The neighborhood sum of absolute differences, d(t, x, y), could bedefined by a sum over a different neighborhood: more than or fewer than8 pixel locations in the strip, and pixel locations from more than onerow could be included.

The threshold T_(max) could be in the range of 50N to 80N where N is thenumber of pixels used in the definition of d(t, x, y) and the luminancedata is in the range 0 to 255. For other data ranges, the thresholdwould be scaled accordingly.

Similarly, the threshold T_(min) could be in the range of 2N to 8N; andthe threshold T_(still) could be in the range of 15N to 35N. Even largerranges may provide useful thresholds.

3A. Pre-filtering Prior to Field Motion Detection Preliminaries

FIGS. 4 a-4 e show notation to help explain the first preferredembodiments and represent a portion of consecutive fields about a pixelto be interpolated. In particular, FIG. 4 c shows the current field attime t=t1 and with the shaded pixel in the center of FIG. 4 c of t=t1 asthe pixel being interpolated; FIG. 4 a shows the second previous field(t=t1−2), FIG. 4 b the previous field (t=t1−1), FIG. 4 d the next field(t=t1+1), and FIG. 4 e the second next field (t=t1+2). Note thealternating parity of the fields in FIGS. 4 a-4 e. The followingdescription provides pseudo-code expression for calculations tointerpolate the shaded (to be interpolated) pixel. The pixels with boldoutline in FIGS. 4 b and 4 d have the same position as the shaded pixelbut are in opposite parity fields. The variables y[i][j][k] representluminance values in the ith field at the (j,k) pixel site with i=0corresponding to t=t1−2, i=1 to t=t1−1, and so forth and the shadedpixel at j=2, k=2. In the following example, when luminance is black,gray, or white, the values of the variable are defined as 0, 0.5, or 1,respectively. That is, the y[i][j][k] variables may be normalized to therange 0 to 1.

First consider a typical simple conventional method for a motiondetector and intra-field and inter-field interpolators as in FIG. 14 ain order to explain the preferred embodiments. Presume theselector/blender block is implemented as a selector. Then theMotion_level for the shaded pixel in the current field (FIG. 4 c) iscalculated by the motion detector using the previous field (FIG. 4 b)and the next field (FIG. 4 d) as follows;

Motion_level = y[1][2][0] − y[3][2][0] + y[1][2][2] − y[3][2][2] + y[1][2][4] − y[3][2][4]When Motion_level>Th_motion, where Th_motion is a threshold value, theselector selects the output of the moving-pixel generator. In thisexplanation, presume that Th_motion=0.

The output of the inter-field interpolator for luminance(pixel_inter_ip_y) is calculated by a field averaging method; i.e.,pixel_inter_ip_(—) y=(y[1][2][2]+y[3][2][2])/2Whereas, the output of the intra-field interpolator for luminance(pixel_intra_ip_y) is calculated by a line averaging method; i.e.,pixel_intra_ip_(—) y=(y[2][2][1])/2

Inter-field and intra-field interpolation processing to calculate huevalues, e.g., Cb (luminance—blue) and Cr (luminance—red), for colorimages are performed in an analogous manner to the processing forluminance.

Using the conventional IP conversion method and notation describedabove, pixel sites in some example images (FIGS. 5 a-5 d) are processedas follows. FIG. 5 a shows original example image sequence, which is apart of whole image area and the image extends to x<0, x>14 and y<0 andy>10. A black still horizontal line runs through the images at the tworows y=4, y=5.

FIG. 5 b shows interlaced video fields of the above original exampleimage, where the odd lines (i.e., y=1, 3, 5, . . .) are removed at t=0,2, 4 (even parity fields) and the even lines are removed at t=1, 3, 5(odd parity fields).

Ideally there is no noise in the input video image and Motion_level=0 atevery pixel site. The IP converted image by the conventional algorithmwith Motion_level=0 is shown in FIG. 5 c. The results for t=0 and 5 aremissing because data of t=−1 and 6 are required to process but notdefined in this example.

However, in reality some noise is in the input video signal. Especially,video images from a video encoding/decoding process will have encodingnoise around the edges of the line. When such noise exists around theline in the sample image, Motion_level increases and the output of themoving-pixel generator is selected if the Motion_level exceedsTh_motion. FIG. 5 d shows the IP converted sample image using the outputof the moving-pixel generator. The luminance of pixels at y=3 (upper theline) is “white at t=odd” and “gray at t=even”, although those arealways white in the original image. The luminance of pixels at y=6(below the line) is “gray at t=odd” and “white at y=even”, althoughalways white in the original. The center black line is on “y=5 at t=odd”and “y=4 at t=even”. Human eyes recognize these artifacts as flickeringof the horizontal line, and no flicker noise in images is preferred. Inaddition to no flicker noise, images with the highest possible spatialresolution are generally preferred.

3B. Pre-filtering Field Motion Detection

The preferred embodiment methods improve accuracy of motion detectioneven if the input video signal has some noise. Additionally, thisimproves spatial resolution for slowly moving objects as explainedbelow.

FIG. 3 a shows a schematic diagram of the preferred embodimentinterlace-to-progressive converter. A pre-filter is added before themotion detector. The pre-filter processes input images for the motiondetector to improve detection accuracy. The pre-filter consists of anintra low-pass filtering that spatially low-pass filters every inputvideo field. The spatial low-pass filtering removes noise because thelow-pass-filtering reduces spatial resolution.

Inter-field interpolation causes a comb-like artifact for a movingobject, but when an object moves slowly (moving speed of ˜1pixel/field), the artifact is not distinguishable. The spatial low-passfiltering masks the slow motion by the spatial resolution reduction andlowers Motion_level. Due to the lowered Motion_level, the inter-fieldinterpolator (which has higher spatial resolution) is selected by theselector/blender block. By this effect, the spatial resolution of aslowly moving object is improved.

A five-field, five-lateral-pixel, five-vertical-line (5×5×5) region isused in this preferred embodiment pre-filtering. The five-vertical-linemeans five lines in progressive format, i.e., in the input interlacedfields, 2 lines and 3 lines are used at “t=t1−2, t1, and t1+2” and“t=t1−1 and t1+1”, respectively. Thus the target pixel to generate byinterpolation is y[2][2][2]. The pre-filtering and motion detectoroperate as follows.

(a) Pre-filter

A two dimensional, 2-pixel horizontal×2-pixel vertical (2×2), low-passfiltering is applied for each field as follows.

for_(h_cnt = 0; h_cnt < 4; h_cnt + +){y^(′)[h_cnt][0] = (y[0][h_cnt][1] + y[0][h_cnt + 1][1] + y[0][h_cnt][3] + y[0][h_cnt + 1][3])/4y^(′)[h_cnt][1] = (y[1][h_cnt][0] + y[1][h_cnt + 1][0] + y[1][h_cnt][2] + y[1][h_cnt + 1][2])/4y^(′)[h_cnt][2] = (y[1][h_cnt][2] + y[1][h_cnt + 1][2] + y[1][h_cnt][4] + y[1][h_cnt + 1][4])/4y^(′)[h_cnt][3] = (y[2][h_cnt][1] + y[2][h_cnt + 1][1] + y[2][h_cnt][3] + y[2][h_cnt + 1][3])/4y^(′)[h_cnt][4] = (y[3][h_cnt][0] + y[3][h_cnt + 1][0] + y[3][h_cnt][2] + y[3][h_cnt + 1][2])/4y^(′)[h_cnt][5] = (y[3][h_cnt][2] + y[3][h_cnt + 1][2] + y[3][h_cnt][4] + y[3][h_cnt + 1][4])/4y^(′)[h_cnt][6] = (y[4][h_cnt][1] + y[4][h_cnt + 1][1] + y[4][h_cnt][3] + y[4][h_cnt + 1][3])/4}The 2×2 filtering is applied to get y′[j][k] and the obtained valueseach represents the data at the center position of the 4 pixels used forthe 2×2 calculation. FIG. 3 b illustrates the y′[j][k] of thelow-pass-filtered pixels. The shaded square (half in each of the y[1][3]and y′[2][3] filtered pixels in the center of FIG. 3 b) shows the targetpixel position being interpolated. Note that fields 0, 2, and 4 each hadtwo lines (one above the target pixel's line, and one below) lowpassfiltered to give pixels in a single line, the line of the target pixel.Contrarily, fields 1 and 3 each had three lines (the target pixel'sline, the line two above the target, and the line two below the target)filtered to give filtered pixels in two lines, one line above and oneline below the target pixel).

(b) Motion Detector

The motion detector calculates the luminance difference between twofields. The output of the motion detector (Motion_level) is calculatedas follows;

Motion_level = y^(′)[0][0] − y^(′)[0][3] + y^(′)[1][0] − y^(′)[1][3] + y^(′)[2][0] − y^(′)[2][3]+      y^(′)[3][0] − y^(′)[3][3] + y^(′)[0][1] − y^(′)[0][4] + y^(′)[1][1] − y^(′)[1][4] + y^(′)[2][1] − y^(′)[2][4] + y^(′)[3][1] − y^(′)[3][4] + y^(′)[0][2] − y^(′)[0][5] + y^(′)[1][2] − y^(′)[1][5] + y^(′)[2][2] − y^(′)[2][5] + y^(′)[3][2] − y^(′)[3][5] + y^(′)[0][3] − y^(′)[0][6] + y^(′)[1][3] − y^(′)[1][6] + y^(′)[2][3] − y^(′)[2][6] + y^(′)[3][3] − y^(′)[3][6]That is, the motion detection simply sums the 4 filtered pixeldifferences from filtered fields 0 and 2, the 8 filtered pixeldifferences from fields 1 and 3, and the 4 filtered pixel differencesfrom filtered fields 2 and 4.3C. Alternatives for Pre-filtering

The preferred embodiments may be modified in various ways whileretaining one or more of the features of pre-filtering fields prior tomotion detection.

For example, weighting the differences used in the motion detection sumto emphasize the pixels closer to the target pixel; varying thefiltering region (number of fields and area about the target pixel) suchas by using a 3×2 filter in each field or decreasing the number offields to 4 by eliminating the t1+2 and the corresponding differences,and so forth.

The interlace-to-progressive converter could blend the still-pixelinterpolation and the moving-pixel interpolation rather than select onewith a blending factor dependent upon the motion level.

Motion could be detected in each color separately (e.g., RGB or YCbCr),rather than as luminance, and sum the color difference for the motionlevel.

The pre-filtering could use 3×3, 5×5, 2×3, et cetera, windows; and coulduse weighting to emphasize or weaken the center pixel; and so forth.

4A. Field Motion Detection with Feedback

Preferred embodiment motion detection utilizes continuity of imagesequences. That is, in a local time interval and in a local area, animage sequence most likely keeps the same state such as “still”,“moving”, “panning”, and so on. Therefore, when the degree of motion ata certain pixel (x0,y0) is calculated, the past value of the degree ofmotion at the same point (x0,y0) can be expected to be an additionalmotion evaluation without expanding the number of fields of storedimages for the purpose of more accurate motion detection.

FIG. 6 a shows a flow for a first preferred embodiment motion detectionmethod together with an interpolation method decision such as would beused in interlace-to-progressive up-conversion processing. In thispreferred embodiment motion detection method, a feedback loop, which iscomposed of an accumulator, a frame memory, a clip, and a motion stopfunction, is inserted after the motion detection previously-described inthe background. The computed degree of motion for pixel (x,y),sum_diff_area[x][y], is added to the value from the feedback loop,mot_feedback1[x][y]. That is,sum_diff_mem[x][y]=sum_diff_area[x][y]+mot_feedback1[x][y](x=0, 1, . . ., IMAGE_SIZE_(—) X−1; y=0, 2, 4, . . . , IMAGE_SIZE_(—) Y−2)or(x=0, 1, . . . , IMAGE_SIZE_(—) X−1; y=1, 3, 5, . . . , IMAGE_SIZE_(—)Y−1)Note that the output value sum_diff_mem[x][y] may cause an overflow dueto the recursive accumulation. Therefore, the output valuesum_diff_mem[x][y] is clipped to a maximum value max_of_sum_diff asfollows:

${{sum\_ diff}{{{\_ clip}\lbrack x\rbrack}\lbrack y\rbrack}} = \left\{ \begin{matrix}{{sum\_ diff}{{{\_ mem}\lbrack x\rbrack}\lbrack y\rbrack}} & \left( {{{sum\_ diff}{{{\_ mem}\lbrack x\rbrack}\lbrack y\rbrack}} < {{max\_ of}{\_ sum}{\_ diff}}} \right) \\{{max\_ of}{\_ sum}{\_ diff}} & ({otherwise})\end{matrix} \right.$The preferred embodiment motion detection output value,sum_diff_clip[x][y], is used as the degree of motion, such as in thedecision about interpolation method for up-conversion, and also isstored in the frame memory in the feedback loop.

For pixel values in the range 0 to 255 (so max_of_sum_diff=255), theinterpolation method decision could be: sum_diff_clip[x][y]≦50 indicatesthe state of (x,y) is still and thus temporal interpolation, andsum_diff_clip[x][y]>50 indicates the state of (x,y) is moving and thusspatial interpolation.

The stored value sum_diff_clip[x][y] is used in the motion detection forthe next field of the same parity, and is adjusted using a constant gainfactor const_a_feedback as follows:mot_feedback[x][y]=sum_diff_clip[x][y]×const_(—) a_feedbackThe gain-adjusted value mot_feedback[x][y] is fed into the motion stopfunction.

Typically, const_a_feedback would be chosen in the range of 0-0.5.Values of const_a_feedback determine the steady state of the recursiveaccumulation: in an image sequence with very small motion,sum_diff_area[x][y] will have steady small values, and in the steadystate sum_diff_clip[x][y]=sum_diff_area[x][y]/(1-const_a_feedback). Thusthe value of const_a_feedback influences the threshold for thestill-versus-moving decision.

In case a sequence of images to be motion-detected is an edit sequence,such as a TV program, a change of image scene happens frequently. When ascene change happens, all absolute values of pixel differences in thetarget field are increased. Therefore, the traditional motion detectionprocessing misjudges all pixels in the target field as having largemotion. In the interlace-to-progressive up-conversion system, themisjudgment loses spatial resolution for the output progressive image.At the time of a scene change, the past motion degree mot_feedback[x][y]is already meaningless and should not be reused. The motion stopfunction effectively judges whether the input value mot_feedback[x][y]should be output or not, depending upon the continuity of imagesequence. If there is expected to be continuity in the current state ofimage sequence, the motion stop function outputs the past motion degreemot_feedback[x][y] directly. Otherwise, the motion stop function cancelsthe feedback factor and outputs zero. To estimate the continuity of theimage sequence, the variation of the summation of absolute values of allpixel differences between the two successive fields of the same paritycan be utilized. For example, define for even and odd fields:

$\left\{ {\begin{matrix}{{{sum\_ of}{{\_ diff}\lbrack 0\rbrack}} = {\sum\limits_{x = 0}^{{{IMAGE\_ SIZE}{\_ X}} - 1}\;{\sum\limits_{{y = 0},2,4,\ldots}^{{{IMAGE\_ SIZE}{\_ Y}} - 2}\;{{{{{{in\_ y}\left\lbrack {k - 2} \right\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} - {{{{in\_ y}\lbrack k\rbrack}\lbrack x\rbrack}\lbrack y\rbrack}}}}}} \\{{{sum\_ of}{{\_ diff}\lbrack 1\rbrack}} = {\sum\limits_{x = 0}^{{{IMAGE\_ SIZE}{\_ X}} - 1}\;{\sum\limits_{{y = 0},2,4,\ldots}^{{{IMAGE\_ SIZE}{\_ Y}} - 2}\;{{{{{{in\_ y}\lbrack k\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} - {{{{in\_ y}\left\lbrack {k + 2} \right\rbrack}\lbrack x\rbrack}\lbrack y\rbrack}}}}}}\end{matrix}{or}\left\{ \begin{matrix}{{{sum\_ of}{{\_ diff}\lbrack 0\rbrack}} = {\sum\limits_{x = 0}^{{{IMAGE\_ SIZE}{\_ X}} - 1}\;{\sum\limits_{{y = 1},3,5,\ldots}^{{{IMAGE\_ SIZE}{\_ Y}} - 1}\;{{{{{{in\_ y}\left\lbrack {k - 2} \right\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} - {{{{in\_ y}\lbrack k\rbrack}\lbrack x\rbrack}\lbrack y\rbrack}}}}}} \\{{{sum\_ of}{{\_ diff}\lbrack 1\rbrack}} = {\sum\limits_{x = 0}^{{{IMAGE\_ SIZE}{\_ X}} - 1}\;{\sum\limits_{{y = 1},3,5,\ldots}^{{{IMAGE\_ SIZE}{\_ Y}} - 1}\;{{{{{{in\_ y}\lbrack k\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} - {{{{in\_ y}\left\lbrack {k + 2} \right\rbrack}\lbrack x\rbrack}\lbrack y\rbrack}}}}}}\end{matrix} \right.} \right.$Using these amount values of pixel difference at the same point in twosuccessive same parity fields, sum_of_diff[0], sum_of_diff[1], and thethreshold th_mot_stop_detection, the output of the motion stop function,mot_feedback1[x][y], can be determined by the following method:

${{tmp\_ sum}{\_ of}{\_ diff}} = \left\{ {{\begin{matrix}{{{sum\_ of}{{\_ diff}\lbrack 0\rbrack}} - {{sum\_ of}{{\_ diff}\lbrack 1\rbrack}}} & \left( {{{sum\_ of}{{\_ diff}\lbrack 0\rbrack}} > {{sum\_ of}{{\_ diff}\lbrack 1\rbrack}}} \right) \\0 & ({otherwise})\end{matrix}{{{mot\_ feedback}\lbrack x\rbrack}\lbrack y\rbrack}} = \left\{ \begin{matrix}{{{mot\_ feedback}\lbrack x\rbrack}\lbrack y\rbrack} & \left( {{{tmp\_ sum} - {of\_ diff}} < {{th\_ mot}{\_ stop}{\_ detection}}} \right) \\0 & ({otherwise})\end{matrix} \right.} \right.$Note that a scene transition from “move” to “still” in which the pastmotion degree should be canceled is on condition that the past amountvalue of pixel difference, sum_of_diff[0], is large and the next amountvalue of pixel difference, sum_of_diff[1], is low. That is, the changeof scene can be detected by measuring sum_of_diff[0]-sum_of_diff[1]. Forluminance pixel values in the range 0 to 255, th_mot_stop_detectionwould typically be chosen to be in the range of 30×(number of pixels ina field) to 50×(number of pixels in a field). And in VGA example(=153600 pixels per field), this would have th_mot_stop_detection in therange 4608000 to 7680000.

The past degree of motion, mot_feedback1[x][y], is derived and used toupdate the current degree of motion, sum_diff_area[x][y], to give thenew degree of motion, sum_diff_mem[x][y], which is then clipped to givethe detection motion degree output.

4B. Example of Field Motion Detection

FIG. 6 b shows an effect of the preferred embodiment motion detectionmethod on the same example situation as shown in FIG. 19 b. As with theprior art motion detection method, the current degree of motion at thetarget pixel (x0,y0) in field#1, sum_diff_area[x0[y0], is zero. However,in the first preferred embodiment motion detection method, the currentdegree of motion sum_diff_area[x0][y0] is added to the past degree ofmotion, mot_feedback1x0][y0]. The value mot_feedback1[x0][y0] for theexample can be estimated by calculating the degree of motion at thetarget pixel (x0,y0) as sum_diff_clip[x0][y0] for field#−1. The twoextracted arrays for the target pixel (x0,y0) in field#−1,mot_y[−2][i][j] and mot_y[0][i][j], are:

$\begin{matrix}{{{{{mot\_ y}\left\lbrack {- 2} \right\rbrack}\lbrack i\rbrack}\lbrack j\rbrack} = 255} & \left( {{i = 0},\ldots\mspace{14mu},4,{j = 0},\ldots\mspace{14mu},2} \right) \\{{{{{mot\_ y}\lbrack 0\rbrack}\lbrack i\rbrack}\lbrack j\rbrack} = \left\{ \begin{matrix}255 & \left( {{i = 0},4} \right) \\0 & ({otherwise})\end{matrix} \right.} & \left( {{j = 0},\ldots\mspace{14mu},2} \right)\end{matrix}$Therefore, the absolute values of differences abs_diff[i][j](i=0,1,2;j=0, . . . , 4) between the two extracted pixel arrays are:

${{{abs\_ diff}\lbrack i\rbrack}\lbrack j\rbrack} = \left\{ \begin{matrix}0 & \left( {{j = 0},4} \right) \\255 & ({otherwise})\end{matrix} \right.$If we take the weight factors as follows with the center elementcorresponding to the pixel location:

${weight\_ factor} = {\begin{bmatrix}1 & 2 & 2 & 2 & 1 \\2 & 4 & 4 & 4 & 2 \\1 & 2 & 2 & 2 & 1\end{bmatrix}\text{/}32}$the weighted absolute array weight_diff[i][j] is:

${weight\_ diff} = \begin{bmatrix}0 & 16 & 16 & 16 & 0 \\0 & 32 & 32 & 32 & 0 \\0 & 16 & 16 & 16 & 0\end{bmatrix}$Thus, the degree of motion at (x0,y0) in field#−1,sum_diff_area[x0][y0], is calculated as follows:

${{sum\_ diff}{{{\_ area}\left\lbrack {x\; 0} \right\rbrack}\left\lbrack {y\; 0} \right\rbrack}} = {{\sum\limits_{i = 0}^{4}\;{\sum\limits_{j = 0}^{2}\;{{{weight\_ diff}\lbrack i\rbrack}\lbrack j\rbrack}}} = 192}$Since the degree of motion at (x0,y0), sum_diff_mem[x0][y0], in field#−1is simply the addition of sum_diff_area[x0][y0] calculated above andmot_feedback1[x0][y0] which is the past motion degree prior to field#−1,sum_diff_mem[x0][y0] can be estimated as follows:sum_diff_mem[x0][y0]=sum_diff_area[x0][y0]+mot_feedback1[x0][y0]=192+mot_feedback[x0][y0]≧192If we take the maximum value max_of_sum_diff=255, then the final valueof the degree of motion at the target pixel (x0,y0) in field#−1,sum_diff_clip[x0][y0], is derived:192≦sum_diff_clip[x0][y0]≦255Let the gain value of the feedback loop const_a_feedback be 0.5, thenthe gain-adjusted value mot_feedback[x0][y0] is in the range:96≦mot_feedback[x0][y0]≦128Since there is no scene change in this example image sequence, we canassume that the motion stop function just outputs the original feedbackdata mot_feedback[x0][y0] directly. Therefore, the past degree of motionat the target pixel (x0,y0) in field#1, mot_feedback1[x0][y0], is alsoin this range:96≦mot_feedback1[x0][y0]≦128Since the current degree of motion at the target pixel (x0,y0) infield#1, sum_diff_area[x0][y0], is zero, the degree of motion at(x0,y0), sum_diff_mem[x0][y0], in field#1 is thus also in this range:96≦sum_diff_mem[x0][y0]≦128And, the final value of the degree of motion at the target pixel (x0,y0)in field#1, sum_diff_clip[x0][y0] is derived again usingmax_of_sum_diff=255:96≦sum_diff_clip[x0][y0]≦128Hence, a non-zero value for the degree of the motion has been derived,and the effectiveness of the preferred embodiment motion detectionmethod using the feedback loop for the degree of motion without anextension of the number of field images to be stored has been shown.4C. Second Preferred Embodiment Field Motion Detection

FIG. 7 a shows the data flow for the second preferred embodiment motiondetection method which is similar to the first preferred embodimentmotion detection with some functional additions for use in a preferredembodiment interlace-to-progressive up-conversion system. First, thenumber of fields to be stored can be upgraded depending on the bandwidthof memory and the size of memory given to this system in the targetapplications. From five stored fields with the target field as field#2,five input data arrays mot_y[k][i][j] are extracted as follows:

${{{{mot\_ y}\lbrack k\rbrack}\lbrack i\rbrack}\lbrack j\rbrack} = \left\{ \begin{matrix}{{{{in\_ y}\lbrack k\rbrack}\left\lbrack {x + i - 2} \right\rbrack}\left\lbrack {y + {2j} - 1} \right\rbrack} & \left( {{k = 0},2,4,{i = 0},\ldots\mspace{14mu},4,{j = 0},1} \right) \\{{{{in\_ y}\lbrack k\rbrack}\left\lbrack {x + i - 2} \right\rbrack}\left\lbrack {y + {2j} - 2} \right\rbrack} & \left( {{k = 1},3,{i = 0},\ldots\mspace{14mu},4,{j = 0},\ldots\mspace{14mu},2} \right)\end{matrix} \right.$

In the next step, a low pass filter is applied to these extracted arraysmot_y[k][i][j] before absolute difference calculations:

$\begin{matrix}{{{mot\_ area}{{{{\_ y}\lbrack k\rbrack}\lbrack i\rbrack}\lbrack j\rbrack}} = \frac{\begin{matrix}{{{{{mot\_ y}\lbrack k\rbrack}\lbrack i\rbrack}\lbrack j\rbrack} + {{{{mot\_ y}\lbrack k\rbrack}\left\lbrack {i + 1} \right\rbrack}\lbrack j\rbrack} +} \\{{{{{mot\_ y}\lbrack k\rbrack}\lbrack i\rbrack}\left\lbrack {j + 1} \right\rbrack} + {{{{mot\_ y}\lbrack k\rbrack}\left\lbrack {i + 1} \right\rbrack}\left\lbrack {j + 1} \right\rbrack}}\end{matrix}}{(4)}} \\{\left( {{k = 0},2,4,{i = 0},\ldots\mspace{14mu},3,{j = 0}} \right)\mspace{14mu}{or}\mspace{14mu}\left( {{k = 1},3,{i = 0},\ldots\mspace{14mu},3,{j = 0},1} \right)}\end{matrix}$This low-pass filter smoothes the input data array mot_y[k][i][j] andcontributes stability to the degree of motion at each target pixel,especially in image sequences which includes high frequency components.The low-pass filtering reduces array sizes; the 5×2 arrays become 4×1and the 5×3 arrays become 4×2.

Absolute values of differences between the low-pass filtered arrays intwo fields are calculated as follows:diff_area_(—) y[k][i][j]=|mot_area_(—) y[k][i][j]−mot_area_(—)y[k+2][i][j]|(k=0,2,i=0, . . . , 3, j=0) or (k=1, i=0, . . . , 3, j=0,1)

As to the derivation of sum_diff_area[x][y], it is done by a simplesummation of all absolute values of differences diff_area_y[k][i][j]without weighting operations for the purpose of design simplification:

${{{sum\_ diff}{{{\_ area}\lbrack x\rbrack}\lbrack y\rbrack}} = \left\lbrack {{\sum\limits_{i = 0}^{3}\;{\sum\limits_{j = 0}^{1}\;{{diff\_ area}{{{{\_ y}\lbrack 1\rbrack}\lbrack i\rbrack}\lbrack j\rbrack}}}} + {\sum\limits_{i = 0}^{3}\;{\sum\limits_{j = 0}^{0}\;\left( {{{diff\_ area}{{{{\_ y}\lbrack 0\rbrack}\lbrack i\rbrack}\lbrack j\rbrack}} + {{diff\_ area}{{{{\_ y}\lbrack 2\rbrack}\lbrack i\rbrack}\lbrack j\rbrack}}} \right)}}} \right\rbrack}\operatorname{>>}4$Note that the shift by 4 bits (divide by 16) compensates for thesummation over a total of 16 terms. In the prior preferred embodimentthe weight factors provided the summation compensation.

Next, subtract the constant const_b from sum_diff_area[x][y]:sum_diff_(—) b[x][y]=sum_diff_area[x][y]−const_(—) b

Depending upon the conditions of the source of input image sequence, theinput image sequence includes many kinds of noise such as temporal noiseby CCD dark current or bad reception conditions for radio wavebroadcasts, and encoded noise by a bit rate reduction in digital videocompression systems. For these bad input source conditions, the absolutevalue of the temporal differences described above increases, so thevalue of degree of motion also tends to be higher than for noiselessinput sources. By subtracting the value const_b, the noise factor in thegiven absolute difference value can be stably disregarded. Typicalvalues for const_b would be in the range 5 to 10.

FIG. 7 b shows the flow for a motion stop function in the feedback loopof a second preferred embodiment motion detection. In this function, thefields of the temporal differences sum_of_diff[k](k=0, 1, 2) areconsidered as a simple extension of the first preferred embodimentmotion stop scheme described above:

${{sum\_ of}{{\_ diff}\lbrack k\rbrack}} = \left\{ \begin{matrix}{{\sum\limits_{x = 0}^{{{IMAGE\_ SIZE}{\_ X}} - 1}\;{\sum\limits_{{y = 0},2,4,\ldots}^{{{IMAGE\_ SIZE}{\_ Y}} - 2}\;{{{{{{{in\_ y}\lbrack k\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} - {{{{in\_ y}\left\lbrack {k + 2} \right\rbrack}\lbrack x\rbrack}\lbrack y\rbrack}}}\mspace{14mu}{or}}}}\;} \\{\sum\limits_{x = 0}^{{{IMAGE\_ SIZE}{\_ X}} - 1}\;{\sum\limits_{{y = 1},3,5,\ldots}^{{{IMAGE\_ SIZE}{\_ Y}} - 1}\;{{{{{{in\_ y}\lbrack k\rbrack}\lbrack x\rbrack}\lbrack y\rbrack} - {{{{in\_ y}\left\lbrack {k + 2} \right\rbrack}\lbrack x\rbrack}\lbrack y\rbrack}}}}} \\\left( {{k = 0},\ldots,2} \right)\end{matrix} \right.$

Using a given threshold value th_mot_stop, this motion stop functionoperates based on the following method:

for(k=0;k<2;k++){  if( sum_of_diff[k]>sum_of_diff[k+1]){   tmpsum_of_diff[k]=sum_of_diff[k]-sum_of_diff[k+1];  }  else{  tmp_sum_of_diff[k]=0;  } } if((tmp_sum_of_diff[0]>th mot stop)II        (tmp_sum_of_diff[1]>th_m_stop)){  is_motion_stop =true; //cancel feedback factor  // for motion detection } else{  is_motion_stop=false; }

Next, apply the clip operation to the addition of the current degree ofmotion, sum_diff_area[x][y], and the past degree of motion from thefeedback loop, mot_feedback1[x][y]. Then the clipped valuesum_diff_clip[x][y] is multiplied by a constant gain const_a:sum_differ_(—) a[x][y]=sum_diff_clip[x][y]×const_(—) aThis operation is to adjust the sensitivity of the given degree ofmotion and is effective to capture the motion, especially in lowcontrast image sequences. Typical values chosen for const_a would be inthe range 1 to 4.

In this way, the preferred embodiment motion detection system outputssum_diff_a[x][y] as the final value of the motion degree at the targetpixel (x,y). And this motion degree is used in the preferred embodimentinterpolation decision for interlace-to-progressive up-conversion.

4D. Motion Detection Alternatives

The preferred embodiments can be modified in various ways whileretaining the feature of a feedback loop for prior field motioninformation to adjust current field motion information.

For example, each of the constants const_a_feedback, const_a, andconst_b could be adaptively adjusted, such as decrease const_a_feedbackwhen one of sum_of_diff[j] is large for several successive fields.

The M×N array size and corresponding matrices weight_factor[i][j] couldbe adaptively adjusted, with smaller arrays used when most pixels have astate of moving.

The motion stop function could extend to the next same parity field asthe scene-change field to eliminate the preceding largesum_diff_clip[x][y] from the frame memory.

The Motion stop function could be extended to include the next field ofthe same parity after a scene change. Consider this simple example (twotables below) of two pixels for up-conversion, one in the even parityfield and one in the odd parity field. Presume each pixel has a state ofstill (e.g., sum_diff_area[x][y] randomly varies about 10) both beforeand after a new scene; the new scene starts at field #8. The first tableis for the first preferred embodiment motion stop function, and thesecond table is for the second preferred embodiment motion stop function(FIG. 7 b).

In the first table the first column is field number, the second columnis sum_diff_area[x][y], the third column is frame memory (equals thepreceding same parity field sum_diff_mem[x][y]), the fourth column ismot_feedback[x][y], the fifth column is sum_of_diff[0]−sum_of_diff[1](denoted just as “large” or “small”), and the sixth column issum_diff_mem[x][y]. Presume gain const_a_feedback=0.5, the scene changegives sum_diff_area[x][y]=100 or 110, and the initial condition forframe memory is 25 for even parity field and 22 for the odd parityfield. Integer arithmetic with round-off is used. The scene change atfield #8 implies: (1) sum_diff_area[x][y] is large only for fields #7and #8, (2) sum_of_diff[0]−sum_of_diff[1] is positive and greater thanthe threshold only for fields #8 and #9, (3) the largemot_feedback[x][y] for field #9 is stopped by Motion stop, but the largemot_feedback[x][y] for field #10 is not stopped by Motion stop.

Field #7 has large sum_diff_mem[x][y] and thus correctly avoids temporalinterpolation (fields#6-#8), field #8 also has large sum_diff_mem[x][y]and thus also correctly avoid temporal interpolation (fields#7-#9), andfield #9 has small sum_diff_mem[x][y] due to motion stop and correctlyuses temporal interpolation of fields #8-#10.

field frame s_d[0] − # s_d_area memory mot_feed s_d[1] s_d_mem 0 10 25(initial) 13 small − small 23 1 12 22 (initial) 11 small − small 23 2 823 12 small − small 20 3 11 23 12 small − small 23 4 9 20 10 small −small 19 5 9 23 12 small − small 21 6 12 19 10 small − large 22 7 100 2111 small − large 111  8 110 22 11 large − small 110 (121) new scene 9 10111  56 large − small 10 (66) 10 12 110  55 small − small 67 11 9 10 5small − small 14 12 11 67 34 small − small 45 13 11 14 7 small − small18 14 8 45 23 small − small 31 15 12 18 9 small − small 21 16 10 31 16small − small 26 17 11 small − small 18 9 small − smallThe same effect of not stopping the large memory for the trailing field#10 also applies with the FIG. 7 b motion stop function as in the secondtable:

frame field mem- mot_ s_d[0] − s_d[1]; # s_d_area ory feed s_d[1] −s_d[2] s_d_mem 0 10 25 13 small − small; small − small 23 (initial) 1 1222 11 small − small; small − small 23 (initial) 2 8 23 12 small − small;small − small 20 3 11 23 12 small − small; small − small 23 4 9 20 10small − small; small − small 19 5 9 23 12 small − small; small − small21 6 12 19 10 small − small; small − large 22 7 100 21 11 small − large;large − large 111  8 110 22 11 large − large; large − small 110 (121)new 9 10 111  56 large − small; small − small 10 (66) 10 12 110  55small − small; small − small 67 11 9 10  5 small − small; small − small14 12 11 67 34 small − small; small − small 45 13 11 14  7 small −small; small − small 18 14 8 45 23 small − small; small − small 31 15 1218  9 small − small; small − small 21 16 10 31 16 small − small; small −small 26 17 11 small − small; small − small 18 9 small − small; small −smallThus, by expanding Motion stop activation to the next same parityfields, better quality of up-conversion image is expected. The firstpreferred embodiments provide the basic concept (e.g., FIG. 6 a) and itsexample (FIGS. 7 a, 7 b) which allows a hardware implementationminimizing gate size and memory bandwidth. However, a generalization ofthe Motion stop function would have features such as:

-   -   1) The function detects a scene transition from “moving” to        “still”.    -   2) The function disables past parameters concerned with a degree        of motion.    -   3) The function activates the function 2) using the function 1)        to a target image or successive images.        5A. Preliminary Field Edge Detection Discussion

FIGS. 9 a-9 c show notation to help explain the problems of the priorart plus the first preferred embodiment methods to overcome theseproblems. In particular, FIGS. 9 a-9 c represent a portion ofconsecutive fields about a pixel to be interpolated. FIG. 9 b shows thecurrent field which correspond to the time t =t1 and with the shadedpixel in the center of FIG. 9 b as the pixel being interpolated; FIG. 9a shows the previous field (t=t1−1), and FIG. 9 c the next field(t=t1+1). Note the alternating parity of the fields in FIGS. 9 a-9 c.The following description provides pseudo-code expression forcalculations to interpolate the shaded (to be interpolated) pixel. Thepixels with bold outline in FIGS. 9 a and 9 c have the same position asthe shaded pixel but are in opposite parity fields. The variable y[j][k]represents the luminance value in the current field at the (j,k) pixelsite with the shaded pixel at j=3, k=3. Analogously, the variablesy_(—i)[j][k] and y_(+i)[j][k] represent luminance values at the (j,k)pixel site in the preceding and next fields, respectively. In thefollowing example, when luminance is black, gray, or white, the value ofa variable is defined as 0, 0.5, or 1, respectively. That is, they[j][k] and y_(±i)[j][k] variables may be normalized to the range 0 to1.

FIG. 14 c shows a schematic diagram of the conventional index generatorfor an edge angle. The difference between the upper and lower pixelvalues in the direction is calculated. The “ABS” block calculates anabsolute value of the difference and outputs as the index. Note that theangle associated with an edge is the direction of the vector normal tothe edge in the coordinate system with x increasing to the right and yincreasing downwards.

Here, three direction detector cases (−45°, 0° and +45°, where 0° meansa vertical edge) are used. Indexes for −45°, 0° and +45° are calculatedas following pseudocode;index_for_(—)−45°=|y[2][2]−y[4][4]|index_for_(—)0°=|y[3][2]−y[3][4]|index_for_(—)+45°=|y[4][2]−y[2][4]In the calculations, when the two pixels in a direction have the sameluminance value or color value, the index becomes 0. Therefore, thedirection determiner chooses the direction that has the smallest indexvalues, i.e.,

index_min = minimum value of index_for_-45°, index_for_0°,         andindex_for_+45° if (index_min = index_for_−45°)  direction = −45° else if(index_min = index_for_0°)  direction = 0° else  direction = +45°

As an example, FIG. 14 d illustrates a vertical thin line (shadedpixels) in a field and the above calculation, where the conventionalmethod causes a problem. The center pixel (with heavy outline) is theone being interpolated. The solid line segments represent absolute valuecalculation of the difference of the two pixel values at the ends of theline segments. The direction determiner can not choose the directioncorrectly because the indexes for all three directions have the samevalue “zero” because the pair of pixel values in each calculation areequal. For the above pseudocode case, “direction =−45°” is output by theedge-direction detector though the edge direction is 0°. Therefore thedirection detector needs to be improved.

The Motion_level is calculated by the motion detector using the previousand next fields using a typical conventional method as follows;

$\begin{matrix}\begin{matrix}{{Motion\_ level} = {{{{{y_{- 1}\lbrack 3\rbrack}\lbrack 1\rbrack} - {{y_{+ 1}\lbrack 3\rbrack}\lbrack 1\rbrack}}} +}} \\{{{{{y_{- 1}\lbrack 3\rbrack}\lbrack 3\rbrack} - {{y_{+ 1}\lbrack 3\rbrack}\lbrack 3\rbrack}}} +} \\{{{{y_{- 1}\lbrack 3\rbrack}\lbrack 5\rbrack} - {{y_{+ 1}\lbrack 3\rbrack}\lbrack 5\rbrack}}}\end{matrix} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$When Motion_level>Th_motion, where Th_motion is a threshold value, theselector selects the output of the moving-pixel generator. In thisexample, Th_motion=0 is used. Of course, noise in the signal wouldsuggest a small positive threshold, such as Motion_level=0.05 fory[j][k] in the range from 0 to 1.

The output of the inter-field interpolator for luminance(pixel_inter_ip_y) is calculated by the field averaging algorithm, i.e.,pixel_inter_ip_(—) y=(y ⁻¹[3][3]+y ₊₁[3][3])/2  (Eq. 2)The output of the intra-field interpolator for luminance(pixel_dir_ip_y) is calculated as follows;if (direction=−45°) pixel_dir_ip_(—) y=(y[2][2]+y[4][4])/2 else if(direction=0°) pixel_dir_ip_(—) y=(y[3][2]+y[3][4])/2 elsepixel_dir_ip_(—) y=(y[4][2]+y[2][4])/2  (Eq. 3)

Intra-field and inter-field interpolation processing to calculate hue,e.g., Cb (luminance-blue) and Cr (luminance-red) values for color imagesare calculated in an analogous manner to the processing for luminance.

FIG. 10 a shows an original sample image, which is a part of whole imagearea and the image extends with white pixels to x<0 in addition to x>14or y>10. A black box is moving from the left side to the right side onthe still white background. The color of the black box is not uniform,though in the figure the color is printed uniformly.

FIG. 10 b shows interlaced video fields of the above original sampleimage, where “odd lines (i.e., y=1, 3, . . .) at t=even” and “even linesat t=odd” are removed.

FIG. 10 c shows the output of the motion detector, where filled pixelsare judged as moving pixels, i.e., Motion_level>Th_motion=0 and theoutput of the intra-field interpolator is selected. At the pixels ofMotion_level≦Th_motion, the output of the inter-field interpolator isused. In order to understand the detected result, note that the blackcolor is not uniform in the area. Motion detector results are not drawnfor y=0 and 1 because data of y=−2 and −1 are required to calculate thembut not defined in this explanation. The results of motion detection fort=0 and 5 are also missing because of no data for t=−1 and 6.

Output of the conventional IP conversion is drawn in FIG. 10 d. Theproblem of the conventional IP converter is line flickering occurs atthe moving lateral edges. Color of pixels at y=5 below the box is {blackat t=odd} and {gray at t=even}, though those are always black in theoriginal image. Color of pixels at y=6 below the box is “gray at t=odd”and “white at y=even”, though always white in the original. The color aty=5 and 6 changes at every progressive frame, i.e., 60 times per second,which human eyes recognize as flickering of the moving lateral edge.

5B. First Preferred Embodiments Field Edge Detection

The preferred embodiment methods detect the edge direction with improvedaccuracy. FIG. 8 a shows a schematic diagram of a first preferredembodiment index generator for an edge angle. Let's consider the angle#0. The index generator for the edge angle #0 consists of the upper andlower pixel pattern generators in the angle #0 and the patterncomparator. The pixel pattern generators generate pixel patternsconsisting of more than a single pixel. The generated pixel patternvaries with the direction being detected; see FIG. 8 e. The output ofthe pattern comparator varies with similarities between the upper andlower patterns.

FIG. 8 b shows a schematic diagram of a preferred embodimentmoving-pixel generator; the moving-pixel generator would be used in anIP converter as in FIG. 14 a. The moving-pixel generator consists of anedge-direction and lateral-edge detector, intra-field interpolator,lateral-edge interpolator and selector/blender blocks. The detectorexamines if the target pixel, i.e., the pixel being interpolated, is ata lateral-edge in addition to the edge-direction detection. The detectorcalculates an index for the probability that the target pixel is on alateral-edge. The lateral-edge interpolator generates a pixel value whenthe pixel is on a lateral-edge. The selector/blender block selects orblends the outputs of intra-field and lateral-edge interpolators. Whenthe lateral-edge probability is high, the output of the lateral-edgeinterpolator is chosen or the blending factor of the lateral-edgeinterpolator is large instead of the output of the intra-fieldinterpolator, and vice-versa.

FIG. 8 c shows a schematic diagram of the first preferred embodimentedge-direction and lateral-edge detector. The “edge-direction andlateral-edge detector” is close to the conventional “edge-directiondetector” but in the “edge-direction and lateral-edge detector” thedirection determiner also outputs a lateral-edge probability index.

FIG. 8 d illustrates the directions Dir1˜Dir7 detected in the firstpreferred embodiment. FIG. 8 e illustrates the pixel patterns. Theshaded center pixel is the one being interpolated. The pixel patternsmarked as “U” and “L” are the pixel patterns generated by the upper andlower pixel pattern generators, respectively. When the upper and lowerpixel patterns do not match, the pattern comparator outputs a largevalue as the index, and vice-versa. In the pattern comparator, theindexes for each direction (sum_dir[]) are calculated as follows;

Index for Dir1:

$\begin{matrix}{{{sum\_ dir}\lbrack 1\rbrack} = {{{{{y\lbrack 0\rbrack}\lbrack 2\rbrack} - {{y\lbrack 3\rbrack}\lbrack 4\rbrack}}} +}} \\{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack} - {{y\lbrack 4\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack} - {{y\lbrack 5\rbrack}\lbrack 4\rbrack}}} +} \\{{{{y\lbrack 3\rbrack}\lbrack 2\rbrack} - {{y\lbrack 6\rbrack}\lbrack 4\rbrack}}}\end{matrix}$

Index for Dir2:

$\begin{matrix}{{{sum\_ dir}\lbrack 2\rbrack} = {{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack} - {{y\lbrack 3\rbrack}\lbrack 4\rbrack}}} +}} \\{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack} - {{y\lbrack 4\rbrack}\lbrack 4\rbrack}}} +} \\{{{{y\lbrack 3\rbrack}\lbrack 2\rbrack} - {{y\lbrack 5\rbrack}\lbrack 4\rbrack}}}\end{matrix}$

Index for Dir3:

$\begin{matrix}{{{sum\_ dir}\lbrack 3\rbrack} = {\left( {{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack} - {{y\lbrack 2\rbrack}\lbrack 4\rbrack}}}/2} \right) +}} \\{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack} - {{y\lbrack 3\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{y\lbrack 3\rbrack}\lbrack 2\rbrack} - {{y\lbrack 4\rbrack}\lbrack 4\rbrack}}} +} \\\left( {{{{{y\lbrack 4\rbrack}\lbrack 2\rbrack} - {{y\lbrack 5\rbrack}\lbrack 4\rbrack}}}/2} \right)\end{matrix}$

Index for Dir4:

$\begin{matrix}{{{sum\_ dir}\lbrack 4\rbrack} = {{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack} - {{y\lbrack 2\rbrack}\lbrack 4\rbrack}}} +}} \\{{{{{y\lbrack 3\rbrack}\lbrack 2\rbrack} - {{y\lbrack 3\rbrack}\lbrack 4\rbrack}}} +} \\{{{{y\lbrack 4\rbrack}\lbrack 2\rbrack} - {{y\lbrack 4\rbrack}\lbrack 4\rbrack}}}\end{matrix}$

Index for Dir5:

$\begin{matrix}{{{sum\_ dir}\lbrack 5\rbrack} = {\left( {{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack} - {{y\lbrack 1\rbrack}\lbrack 4\rbrack}}}/2} \right) +}} \\{{{{{y\lbrack 3\rbrack}\lbrack 2\rbrack} - {{y\lbrack 2\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{y\lbrack 4\rbrack}\lbrack 2\rbrack} - {{y\lbrack 3\rbrack}\lbrack 4\rbrack}}} +} \\\left( {{{{{y\lbrack 5\rbrack}\lbrack 2\rbrack} - {{y\lbrack 4\rbrack}\lbrack 4\rbrack}}}/2} \right)\end{matrix}$

Index for Dir6:

$\begin{matrix}{{{sum\_ dir}\lbrack 6\rbrack} = {{{{{y\lbrack 3\rbrack}\lbrack 2\rbrack} - {{y\lbrack 1\rbrack}\lbrack 4\rbrack}}} +}} \\{{{{{y\lbrack 4\rbrack}\lbrack 2\rbrack} - {{y\lbrack 2\rbrack}\lbrack 4\rbrack}}} +} \\{{{{y\lbrack 5\rbrack}\lbrack 2\rbrack} - {{y\lbrack 3\rbrack}\lbrack 4\rbrack}}}\end{matrix}$

Index for Dir7:

$\begin{matrix}{{{sum\_ dir}\lbrack 7\rbrack} = {{{{{y\lbrack 3\rbrack}\lbrack 2\rbrack} - {{y\lbrack 0\rbrack}\lbrack 4\rbrack}}} +}} \\{{{{{y\lbrack 4\rbrack}\lbrack 2\rbrack} - {{y\lbrack 1\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{y\lbrack 5\rbrack}\lbrack 2\rbrack} - {{y\lbrack 2\rbrack}\lbrack 4\rbrack}}} +} \\{{{{y\lbrack 6\rbrack}\lbrack 2\rbrack} - {{y\lbrack 3\rbrack}\lbrack 4\rbrack}}}\end{matrix}$

FIG. 8 f illustrates the above calculation. The shaded center pixel isthe one being interpolated. The solid and dotted lines representabsolute calculation of the difference. The dotted lines mean theabsolute values are divided by two.

Basically, three pixels on the upper and lower lines are used as thepixel pattern. For Dir3 and Dir5, the upper and lower three pixelpattern has some offset from the center position, so a 4-pixel patternis used and the absolute values from the outer pixels are divided by twoto be equivalent to a three pixel pattern. For Dir1 and Dir7, a 4-pixelpattern is also used to include the two pixels adjacent the targetpixel, and dividing by 2 is not performed in order to clearly detect thevertical thin line illustrated in FIGS. 8 g-8 h. FIG. 8 g showsapplication of the index generator for Dir1. Note that if i=2, 3, . . ., 6, then sum_dir[i] lies in the range from 0 to 3 for y[j][k] in therange from 0 to 1, and if i=1 or 7, then sum_dir[i] is in the range 0 to4. In the case of the thin vertical line in FIG. 8 g, sum_dir[4] shouldbe small and sum_dir[i] (i=1-3 and 5-7) should be sufficiently largerthan sum_dir[4]. For FIG. 8 g, sum_dir[4]=0 and sum_dir[i]=2 for i=1-3and 5-7.

Also, a two-line pattern (two pixels each in upper and lower patterns)does not work for the vertical thin line as shown in FIG. 8 h. With atwo-line pattern, the index for Dir1, Dir4 and Dir7 would all have thesame index value (i.e., =0).

In general, the closer to the horizontal line the edge is, the largerpattern, i.e., more pixel width, is required to detect the edge.

The direction determiner decides the direction as follows;

sum_dir_min = minimum value of sum_dir[1], . . . , sum_dir[7]if  (sum_dir_min = sum_dir[4])  direction = Dir4 else if (sum_dir_min =sum_dir[5])  direction = Dir5 else if (sum_dir_min = sum_dir[3]) direction = Dir3 else if (sum_dir_min = sum_dir[6])  direction = Dir6else if (sum_dir_min = sum_dir[2])  direction = Dir2 else if(sum_dir_min = sum_dir[7])  direction = Dir7 else  direction = Dir1

In this embodiment, the intra-field interpolator is implemented asfollows;

if (direction = Dir1)  ave_dir_y = ((y[1][2] + y[4][4])/2 + (y[2][2] +y[5][4])/2)/2 else if (direction = Dir2)  ave_dir_y = (y[2][2]+y[4][4])/2 else if (direction = Dir3)  ave_dir_y = ((y[2][2] +y[3][4])/2 + (y[3][2] + y[4][4])/2)/2 else if (direction = Dir4) ave_dir_y = (y[3][2] +y[3][4])/2 else if (direction = Dir5)  ave_dir_y= ((y[3][2] + y[2][4])/2 + (y[4][2] + y[3][4])/2 else if (direction =Dir6)  ave_dir_y_8[6] = (y[4][2] + y[2][4])/2 else // direction = Dir7 ave_dir_y = ((y[4][2] + y[l][4])/2 + (y[5][2] + y[2][4])/2)/2pixel_dir_ip_y = ave_dir_yThe pixel values on the upper and lower lines in the detected directionare averaged. FIG. 8 i illustrates the above calculation. The solid anddotted lines mean averaging the two pixel values. The dotted lines meanthe two averaged values are averaged again (second averaging) becausethe center of the averaged two pixels does not align to the center ofthe target pixel being interpolated. To eliminate the offset, the secondaveraging is performed.

Interpolation processing to calculate hue, e.g., Cb (luminance—blue) andCr (luminance—red) values for color images is done in an analogousmanner to the processing for luminance using a direction detected byluminance data.

All functions of the preferred embodiment edge-direction detectordescribed above are included in the preferred embodiment edge-directionand lateral-edge detector of FIG. 8 c. The direction determiner has anadditional function, i.e., a lateral-edge probability index iscalculated. And “sum_dir_min” is used for the index in this embodiment,i.e., the difference between the upper and lower pixel patterns; recallthat sum_dir_min lies in the range from 0 to 3 for y[j][k] in the rangefrom 0 to 1.

Intra-field interpolation causes flicker noise as described above, sointer-field interpolation is used for the lateral-edge interpolation.The output of the lateral-edge interpolator for luminance(pixel_lateral_ip_y) is calculated by the following field averaging overtime algorithm,pixel_lateral_ip_y=(y ⁻¹[3][3]+y ₊₁[3][3])/2Cb (luminance—blue) and Cr (luminance—red) values for color images arecalculated in an analogous manner to the processing for luminance.

When a selector is implemented in the selector/blender block, theluminance output of the moving pixel generator (out_mov_ip_y) isdetermined as follows;if (sum_dir_min<Th_lateral_edge) out_mov_ip_(—) y=pixel_dir_ip_(—) yelse if (sum_dir_min=Th_lateral_edge) out_mov_ip_(—) y=pixel_dir_ip_(—)y or pixel_lateral_ip_(—) y (which depends on implementation) elseout_mov_ip_(—) y=pixel_lateral_ip_(—) ywhere pixel_dir_ip_y is the output of the inter-field interpolator forluminance.

When a blender is implemented in the selector/blender block, the blenderworks as follows to generate the output of the moving pixel generator.The blending factor (blendingFactor) is generated using the functionillustrated in FIG. 8 j: “thA” is the slope and “thB” is the intercept.“thA” and “thB” are the tuning parameters for the blending. For example,“thA=1” and “thB=24” work well for 8-bit logic (i.e., when y[j][k] liesin the range 0 to 255, sum_dir_min/4 is used as the horizontal axis(because the maximum value of sum_di_min is 4*255), andblendingFacor=255 represents 100%).

The outputs of the intra-field and lateral-edge interpolators areblended as follows;out_mov_ip_(—) y=pixel_dir_ip_(—)y*(100−blendingFactor)/100+pixel_lateral_ip_(—) y*(blendingFactor)/100where blendingFactor is expressed in percent.

Cb (luminance—blue) and Cr (luminance—red) values for color images arecalculated in an analogous manner to the processing for luminance.

The previous interlaced sample image shown in FIGS. 10 a-10 d is usedhere to explain the effect of the preferred embodiment. The conventionalmotion detector and still pixel generator described in the foregoing areused. A selector is used for the selector/blender block in the movingpixel generator instead of a blender. Th_lateral_edge is low enough forthe selector to select the output of the intra-field interpolator onlywhen sum_dir_min is less than or equal to 4 times “variation of thefilled black colors of the sample image”. In this example, variation ofthe filled black colors of the sample image are enough smaller than thecolor difference between black (0) and gray (0.5).

FIGS. 11 a-11 b shows the results of the motion detection andlateral-edge detection. The pixels marked as “M” are the ones that areMotion_level is not 0 and are not on the lateral-edge, i.e., the outputof the intra-field interpolator is selected as that of the IP converter.The filled pixels are Motion_level is not 0 and are on the lateral-edge,i.e., the output of the lateral-edge interpolator is selected as that ofthe IP converter.

At t=even, the IP converted image is same as the original. At t=odd, theerror from the original progressive source is only the bottom cornerpixels of the moving box. Thus the preferred embodiment greatly reducesflicker noise at the moving lateral edges.

6A. Uniformity Blending Preliminary Discussion

FIGS. 4 a-4 e show notation to help explain prior art problems pluspreferred embodiment methods to overcome these problems. In particular,FIGS. 4 a-4 c represent a portion of consecutive fields about a pixel tobe interpolated: FIG. 4 c shows the current field at time t=t1 and withthe shaded pixel in the center of FIG. 4 c of t=t1 as the pixel beinginterpolated; FIG. 4 a shows the second previous field (t=t1−2), FIG. 4b the previous field (t=t1−1), FIG. 4 d the next field (t=t1+1), andFIG. 4 e the second next field (t=t1 +2). Note the alternating parity ofthe fields in FIGS. 4 a-4 e. The following description providespseudo-code expression for calculations to interpolate the shaded (to beinterpolated) pixel. The pixels with bold outline in FIGS. 4 b and 4 dhave the same position as the shaded pixel but are in opposite parityfields. The variables y[i][j][k] represent luminance values in the ithfield at the (j,k) pixel site with i=0 corresponding to t=t1−2, i=1 tot=t1−1, and so forth, and the shaded pixel at j=2, k=2 is theinterpolation target. In the following example, when luminance is black,gray, or white, the values of the variable are defined as 0, 0.5, or 1,respectively. That is, the y[i][j][k] variables may be normalized to therange 0 to 1.

FIG. 14 e is a schematic diagram of a conventional motion detector. Whenthe current field is a top-field, the opposite field is a bottom-field,and vice-versa. Motion levels calculated from the current and oppositefields are summed up and output to the blender as Motion_level.

The motion detector calculates the color difference between two fields.The outputs of the Motion_level calculator from the current(Motion_level_c) and opposite (Motion_level_o) fields are calculated asfollows;

$\begin{matrix}{{{Motion\_ level}{\_ c}} = {{{{{{y\lbrack 0\rbrack}\lbrack 0\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 0\rbrack}\lbrack 1\rbrack}}} +}} \\{{{{{{y\lbrack 0\rbrack}\lbrack 1\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 1\rbrack}\lbrack 1\rbrack}}} +} \\{{{{{{y\lbrack 0\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack}}} +} \\{{{{{{y\lbrack 0\rbrack}\lbrack 3\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 3\rbrack}\lbrack 1\rbrack}}} +} \\{{{{{{y\lbrack 0\rbrack}\lbrack 4\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 4\rbrack}\lbrack 1\rbrack}}} +} \\{{{{{{y\lbrack 0\rbrack}\lbrack 0\rbrack}\lbrack 3\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 0\rbrack}\lbrack 3\rbrack}}} +} \\{{{{{{y\lbrack 0\rbrack}\lbrack 1\rbrack}\lbrack 3\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 1\rbrack}\lbrack 3\rbrack}}} +} \\{{{{{{y\lbrack 0\rbrack}\lbrack 2\rbrack}\lbrack 3\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 3\rbrack}}} +} \\{{{{{{y\lbrack 0\rbrack}\lbrack 3\rbrack}\lbrack 3\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 3\rbrack}\lbrack 3\rbrack}}} +} \\{{{{{y\lbrack 0\rbrack}\lbrack 4\rbrack}\lbrack 3\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 4\rbrack}\lbrack 3\rbrack}}}\end{matrix}$ $\begin{matrix}{{{Motion\_ level}{\_ o}} = {{{{{{y\lbrack 1\rbrack}\lbrack 0\rbrack}\lbrack 0\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 0\rbrack}\lbrack 0\rbrack}}} +}} \\{{{{{{y\lbrack 1\rbrack}\lbrack 1\rbrack}\lbrack 0\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 1\rbrack}\lbrack 0\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 0\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 2\rbrack}\lbrack 0\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 3\rbrack}\lbrack 0\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 3\rbrack}\lbrack 0\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 4\rbrack}\lbrack 0\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 4\rbrack}\lbrack 0\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 0\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 0\rbrack}\lbrack 2\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 1\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 1\rbrack}\lbrack 2\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 3\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 3\rbrack}\lbrack 2\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 4\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 4\rbrack}\lbrack 2\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 0\rbrack}\lbrack 4\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 0\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 1\rbrack}\lbrack 4\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 1\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 4\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 2\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 3\rbrack}\lbrack 4\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 3\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{y\lbrack 1\rbrack}\lbrack 4\rbrack}\lbrack 4\rbrack} - {{{y\lbrack 3\rbrack}\lbrack 4\rbrack}\lbrack 4\rbrack}}}\end{matrix}$

When the current field being interpolated is the top-field (i.e., bottomlines are being interpolated), Motion_level_c is calculated from the toplines and Motion_level_o is calculated from the bottom lines. When thecurrent field is the bottom-field, Motion_level_c is calculated from thebottom lines and Motion_level_o is calculated from the top lines.

The output of the motion detector (Motion_level) is calculated asfollows;Motion_level=Motion_level_(—) c+Motion_level_(—) oMotion_level_c and Motion_level_o are summed up, i.e., Motion_level iscalculated from both the top-field and the bottom-field.

The output of the inter-field interpolator for luminance(pixel_inter_ip_y) is calculated by field averaging method, i.e.,pixel_inter_ip_(—) y=(y[1][2][2]+y[3][2][2])/2

The output of the intra-field interpolator for luminance(pixel_intra_ip_y) is calculated by line averaging method, i.e.,pixel_intra_ip_(—) y=(y[2][2][1]+y[2][2][3])/2

Inter-field and intra-field interpolation processing to calculate hue,e.g., Cb (luminance—blue) and Cr (luminance—red) values for color imagesare performed in an analogous manner to the processing for luminance.

The luminance output of the blender, i.e., the luminance output of theIP converter is pixel_ip_y.

The blending factor for the blender (intra_fraction), which representsfraction of the output of the intra-field interpolator in the blenderoutput, is generated using the function illustrated in FIG. 14 f. “thB”and “thC” are tuning parameters; typical values for “thB” and “thC”could be in the ranges 0 to 16 and 8 to 32, respectively, for 8-bitlogic where y[i][j][k] lies in the range 0 to 255.

The outputs of the inter-field and intra-field interpolators are blendedas follows;pixel_ip_(—) y=pixel_inter_ip_(—) y*(100−intra_fraction)/100+pixel_intra_ip_(—) y*(intra_fraction)/100where intra_fraction is expressed in percent.

Cb (luminance—blue) and Cr (luminance—red) values for color images arecalculated in an analogous manner to the processing for luminance.

Using the conventional IP conversion method and notation describedabove, pixel sites in several sample images are processed as follows.The parameters of the blender are set as follows;thB=0thC=13

FIG. 13 a shows sample image-A in which a black box is moving from theleft to the right side. FIG. 13 b shows the interlaced sample image-A.Top-fields at t=odd and bottom-fields at t=even. The pixel marked with“A” (pixel-A) is the one being interpolated.Motion_level=13pixel_inter_ip_(—) y=1 (white)pixel_intra_ip_(—) y=0 (black)intra_fraction=100%pixel_ip_(—) y=0 (black)The luminance output of the IP converter (pixel_ip_y) is pixel_ip_y) is0 (black) that matches the original value at the pixel-A, i.e., the IPconversion works well for this case.

FIG. 13 c shows sample image-B in which a black fragment is moving fromthe left to the right side; and FIG. 13 d shows the interlace fields.Motion_level=4pixel_inter_ip_(—) y=1 (white)pixel_intra_ip_(—) y=0 (black)intra_fraction=31%pixel_ip_(—) y=0.69The original luminance value of the pixel-B is 0 (black) butpixel_ip_y=0.69, i.e., the IP conversion does not work well in thiscase.

FIG. 13 e shows an sample image-C in which a gray fragment is movingfrom the left to the right side; and FIG. 13 f shows the interlacefields.Motion level =6.5pixel_inter_ip_(—) y=1 (white)pixel_intra_ip_(—) y=0.5 (gray)intra_fraction=50%pixel_ip_(—) y=0.75The original luminance value of the pixel-B is 0.5 (gray) butpixel_ip_y=0.75 which has about 50% error from the original, i.e., theIP conversion does not work well in this case, either.

As shown above, around a small pixel pattern or a low contrast area,Motion_level detected by the conventional motion detector is not highenough though the target pixel is a part of a moving object and thewrong color is interpolated. If “thB” and “thC” are reduced,intra_fraction becomes 100%, the pixel_intra_y is output, and theinterpolation by the wrong color can be avoided. But by themodification, pixel_intra_y that has lower spatial resolution is used atpixel sites having low Motion_level and causes flickering noise at highfrequency pattern areas.

6B. Uniformity Blending First Preferred Embodiments

The first preferred embodiments interpolate correct color around a smallpixel pattern or a low contrast area plus keep image quality at pixelshaving low motion levels.

FIG. 12 a shows a schematic diagram of the preferred embodimentinterlace-to-progressive converter. A uniformity detector is added tothe conventional IP converter. The uniformity detector detectsuniformity around the target pixel. The output of the detector(intra_diff) is sent to the blender and controls the blending function,i.e., the function to generate intra_fraction.

FIG. 12 b shows a schematic diagram of the uniformity detector, whichconsists of the “color uniformity calculator in the current field”,“color uniformity calculator in the opposite field” and “selector”. Thecolor uniformity calculators detect the variation of color in the areaused for the motion detector. The calculation is performed in thecurrent and opposite field, i.e., both top and bottom fields because themotion detector detects motion using the current and opposite fields.The selector selects the outputs of the color uniformity calculator inthe current and opposite fields.

FIG. 12 c illustrates how the function to generate intra_fraction works.The slope (thA=100/(thC-thB)) varies (by varying thC) with the output ofthe uniformity detector intra_diff. When intra_diff is low, i.e., thecolor of the area is uniform or contrast is low, the slope becomes steepand the blender gets sensitive to Motion_level, i.e., the fraction ofthe output of the intra-field interpolator becomes higher with smallerMotion_level.

The disadvantage of the present approach is that at still (i.e.,Motion_level is low) areas having little pattern or low contrast, theoutput of the moving-pixel generator tends to be selected instead ofthat of still-pixel generator. The output of the still-pixel generatorhas a higher resolution than that of the moving-pixel generator. But atsuch areas, the resolution difference between the outputs of the twopixel generators is not distinguishable.

The color (luminance) uniformity calculator sums up the differences ofcolor in a field. When there is no edge or the patterns have lowcontrast in the area used for the motion detection, the output of theuniformity detector (intra_diff) becomes low.

The output of the color uniformity calculator in the opposite field(intra_diff_pre) is calculated as follows;

$\begin{matrix}{{{intra\_ diff}{\_ pre}} = {{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 0\rbrack}\lbrack 0\rbrack}}} +}} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 1\rbrack}\lbrack 0\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 0\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 3\rbrack}\lbrack 0\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 4\rbrack}\lbrack 0\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 0\rbrack}\lbrack 2\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 1\rbrack}\lbrack 2\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 3\rbrack}\lbrack 2\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 4\rbrack}\lbrack 2\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 0\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 1\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 3\rbrack}\lbrack 4\rbrack}}} +} \\{{{{{y\lbrack 1\rbrack}\lbrack 2\rbrack}\lbrack 2\rbrack} - {{{y\lbrack 1\rbrack}\lbrack 4\rbrack}\lbrack 4\rbrack}}}\end{matrix}$

The output of the color uniformity calculator in the current field(intra_diff_now) is calculated as follows;

$\begin{matrix}{{{intra\_ diff}{\_ now}} = {{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 0\rbrack}\lbrack 1\rbrack}}} +}} \\{{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 1\rbrack}\lbrack 1\rbrack}}} +} \\{{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 3\rbrack}\lbrack 1\rbrack}}} +} \\{{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 4\rbrack}\lbrack 1\rbrack}}} +} \\{{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 0\rbrack}\lbrack 3\rbrack}}} +} \\{{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 1\rbrack}\lbrack 3\rbrack}}} +} \\{{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 3\rbrack}}} +} \\{{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 3\rbrack}\lbrack 3\rbrack}}} +} \\{{{{{y\lbrack 2\rbrack}\lbrack 2\rbrack}\lbrack 1\rbrack} - {{{y\lbrack 2\rbrack}\lbrack 4\rbrack}\lbrack 3\rbrack}}}\end{matrix}$

FIG. 12 d illustrates the above calculations; the shaded pixel is thetarget pixel being interpolated and the pixel with bold outline is thesame position with the shaded pixel but on the previous field. Thedotted lines represent absolute calculation of the difference of twopixel values.

The selector works as follows;intra_diff_now15=intra_diff_now×1.5 if (intra_diff_pre<intra_diff_now15)intra_diff=intra_diff_pre else intra_diff=intra_diff_now15

intra_diff_now15 is used instead of intra_diff_now because the number ofpixels used for t=t1−1 is larger than that for t=t1. The selectorselects the smaller output from the two color uniformity calculators,i.e., if intra_diff_pre or intra_diff_now15 is low, the slope of thefunction to generate intra_diff gets steep.

The blender works as follows;if (intra_diff≦th1_uniform_area) thA=4*thA0 else if(intra_diff≦th2_uniform_area) thA=3*thA0 else if(intra_diff≦th3_uniform_area) thA=2*thA0 else thA=thA0where th1_uniform_area, th2_uniform_area, and th3_uniform_area arethreshold values, th1_uniform_area<th2_uniform_area<th3_uniform_area;and thA0 is the slope when intra_diff is large enough.

When natural images are IP converted, thA0=4, thB=0, th1_uniform_area=4, th2_uniform_area=8, th3_uniform_area=12 work well for 8-bit logicwhen the maximum value of Motion_level=255, the maximum value ofintra_diff=255, and intra_fraction=255 represents 100%.

The thresholds th1_uniform_area=1, th2_uniform_area=2 andth3_uniform_area=3 are used for the sample images. And thA0=100% /13corresponds to the thC described above. The preferred embodiment worksfine for the sample images image-A, image-B, and image-C as shown below.

The calculated value of intra_diff for pixel-A of the sample image-A ofFIGS. 13 a-13 b is 6. The thA does not change from thA0 but this isacceptable because image-A is well IP converted by the conventionalmethod.

The calculated value of intra_diff for pixel-B of the sample image-B ofFIGS. 13 c-13 d is 1.thA=4*thA0thC=3.25intra_fraction=100%pixel_ip_(—) y=0 (black)pixel_ip_y matches the original color (black).

The calculated value of intra_diff for pixel-C of the sample image-C ofFIGS. 13 e-13 f is 3.thA=2×thA0thC=6.5intra fraction=100%pixel_ip_y=0 (black)pixel_ip_y matches the original color (black). Thus the preferredembodiment method overcomes the problems of the conventional.6D. Modifications

The preferred embodiments may be modified in various ways whileretaining the feature of adjusting the blending function for the interand intra interpolation according to a uniformity of the color(luminance) in an area about the target pixel being interpolated.

For example, the thresholds could all be varied; the tuning parameterthB could be varied in a range of −8 to 8; the tuning parameter thA0could be varied in a range of 2 to 12; the tuning parameterth1_uniform_area cold be varied in a range of 2 to 8; the tuningparameter th2_uniform_area could be varied in a range of 3 to 16; thetuning parameter th3_uniform_area could be varied in a range of 4 to 16;the number of selections for thA determination by intra_diff might bereduced or increased; thA could be changed according to the followingfunction:if (intra_diff<thM0) thA=thM1; else if (thM2<intra_diff) thA=thM3; elsethA=(thM3−thM1)/(thM2−thM0)*intra_diff+thM1where typical threshold values for 8-bit data are:thM0=0 (0 to 8)thM1=16 (8 to 64)thM2=12 (4 to 16)thM3=4 (2 to 32);the shape of the piecewise-linear curve for intra_fraction could bechanged to a non-linear curve; the selector might be removed forsimplification and sum up the output of the color uniformity calculatorsfor the current and opposite fields; one of the two color uniformitycalculators can be removed for simplification or when more fields thantwo are used.7. Further Modifications

One or more of the preferred embodiment features (3-2 pulldowndetection, pre-filtering, motion detection with feedback, edge detectionwith patterns, and blending of moving pixel and still pixelinterpolations using uniformity) can be combined into new preferredembodiments.

Further modifications include the tuning parameters for blendingFactorcould be varied and the even the shape of the piecewise linear curvecould be changed, such as to a non-linear curve; the lines used forpixel patterns could be extended to upper pixels from two lines abovethe target and lower pixels from two lines below the target; the numberof edge directions detected could be increased, such as to 9 with pixelpatterns in a 9×2 array; the number of horizontal pixels of thegenerated patterns could be changed, such as increased from 3 pixels forDir2-6 to 4 pixels; and so forth

1. A method of 3-2 pulldown detection executed by a processor,comprising: (a) applying a low pass filter on an input image andcomputing, via the processor, at least one difference between successivefields of the same parity using the maximum of the absolute pixeldifferences averaged over a small number of pixels; (b) comparing saiddifferences for a sequence of differences for determining repeatedfields by utilizing a safety coefficient and excluding still fields fromthe determination, wherein the safety coefficient is implementedutilizing a multiplication operation; (c) when the maximum of two ofsaid differences offset by a predetermined threshold that is less thanthe minimum of the other of said differences, determining that saidinput sequence of fields is a 3-2 pulldown sequence.
 2. A method of aprocessor for motion detection in a sequence of fields, comprising: (a)for a pixel (x,y) not in a first field, computing a difference betweenthe values of pixels in a neighborhood of (x,y) in the field precedingsaid first field and the values of corresponding pixels in the fieldfollowing said first field; (b) computing a motion degree for at leastone pixel in said first field by combining said difference from (a) witha motion degree for said pixel from a second field, said second field ofthe same parity as said first field and preceding said first field,wherein said motion detection pre-filters fields about said target pixeland compares differences of filtered pixels in said filtered fields. 3.A method of a processor for interlace-to-progressive conversion,comprising: (a) providing a still-pixel interpolation and a moving-pixelinterpolation for a target pixel to be interpolated utilizing an edgedirection angle detection; (b) blending said still-pixel interpolationand said moving-pixel interpolation with a factor which depends uponboth a motion level at and a uniformity about said target pixel, whereinthe moving pixel interpolation uses the edge direction angle detectionmethod, comprising: generating an upper pattern of pixels in a field,said upper pattern including at least three pixels in a line above apixel to be interpolated; generating a lower pattern of pixels in saidfield, said lower pattern including at least three pixels in a linebelow said pixel to be interpolated; computing a sum of differences ofpixels in said upper pattern and pixels in said lower pattern;generating a second upper pattern of pixels in a field, said secondupper pattern including at least three pixels in a line above said pixelto be interpolated; generating a second lower pattern of pixels in saidfield, said second lower pattern including at least three pixels in aline below said pixel to be interpolated; computing a sum of differencesof pixels in said second upper pattern and pixels in said second lowerpattern; and determining an edge direction at said pixel to beinterpolated utilizing said sum of differences of pixels in said secondupper pattern and pixels in said second lower pattern.